PaperlayerView exact revision on arXiv

Source-linked AI summary

Models of human preference for learning reward functions

W. Bradley Knox, Stephane Hatgis-Kessell, Serena Booth, Scott Niekum, Peter Stone, Alessandro Allievi

arXiv:2206.02231v3cs.LGcs.AIcs.SYeess.SY

TL;DR

Reward learning from human preferences is often based on partial return, which can miss information about segment desirability. This paper proposes regret-based preferences and finds theoretical identifiability and empirical advantages over partial return, including with human preferences.

  • Problem

    Common RLHF models assume preferences depend only on a segment’s partial return, ignoring start- and end-state values and potentially favoring lucky suboptimal actions.

    Source: §1 · p. 2

  • Method

    The paper models preferences using each trajectory segment’s regret, its deviation from optimal decision-making, rather than only its partial return.

    Source: §1 · p. 2

  • Results

    Regret-based learning identifies an equivalent set of optimal policies with infinite exhaustive preferences and, empirically, better predicts human preferences and learns better-aligned policies than partial return.

    Source: §1 · p. 2, §7 · p. 16

  • Takeaways & Limitations

    The choice of preference model impacts learned reward-function alignment, and regret-based preferences improve on the partial-return model in the reported evaluations.

    Source: §7 · p. 16

    The regret model assumes people can identify optimal and suboptimal segments, an assumption that may be less true in domains where they have less expertise.

    Source: §2.2 · p. 5

Abstract

from arXiv · show

The utility of reinforcement learning is limited by the alignment of reward functions with the interests of human stakeholders. One promising method for alignment is to learn the reward function from human-generated preferences between pairs of trajectory segments, a type of reinforcement learning from human feedback (RLHF). These human preferences are typically assumed to be informed solely by partial return, the sum of rewards along each segment. We find this assumption to be flawed and propose modeling human preferences instead as informed by each segment's regret, a measure of a segment's deviation from optimal decision-making. Given infinitely many preferences generated according to regret, we prove that we can identify a reward function equivalent to the reward function that generated those preferences, and we prove that the previous partial return model lacks this identifiability property in multiple contexts. We empirically show that our proposed regret preference model outperforms the partial return preference model with finite training data in otherwise the same setting. Additionally, we find that our proposed regret preference model better predicts real human preferences and also learns reward functions from these preferences that lead to policies that are better human-aligned. Overall, this work establishes that the choice of preference model is impactful, and our proposed regret preference model provides an improvement upon a core assumption of recent research. We have open sourced our experimental code, the human preferences dataset we gathered, and our training and preference elicitation interfaces for gathering a such a dataset.

1 Introduction

Reward learning from human preferences can help align reinforcement learning with stakeholder interests, but common preference models use only each segment’s partial return. The paper proposes regret-based preferences and reports theoretical identifiability and empirical advantages over partial return.

1 Introduction

  • Misaligned reward functions limit reinforcement learning’s utility and may risk financial cost or human injury or death.Recent RLHF work commonly learns rewards from preferences between pairs of trajectory segments.Source: §1 · p. 2
  • Partial-return preferences ignore segment start and end values and can favor suboptimal actions with lucky outcomes.The paper gives buying a lottery ticket as an example.Source: §1 · p. 2
  • The paper proposes modeling preferences by each segment’s regret, a measure of deviation from optimal decision-making.The models can differ in constant-reward-until-the-end domains, including competitive games and tasks that minimize time to a goal.Source: §1 · p. 2
  • With infinite, exhaustive preferences, regret-based reward learning identifies a reward function with the same set of optimal policies, while partial return lacks this guarantee in multiple contexts.The paper identifies three contexts for the partial-return result: noiseless preferences, variable-length trajectories from a state, and one-transition segments.Source: §1 · p. 2
  • Regret-based learning better predicts collected human preferences and produces better-aligned policies, while also tending to outperform partial return with synthetic preferences.The paper concludes that preference-model choice impacts the alignment of learned reward functions.Source: §1 · p. 2

2 Preference models for learning reward functions

This section describes reward learning from pairwise trajectory-segment preferences and contrasts a partial-return model with a regret-based alternative. The regret model emphasizes decision quality, with stated assumptions about how humans judge segments.

2 Preference models for learning reward functions

  • The task is modeled as an MDP with states, actions, transitions, a discount factor, start-state distribution, and reward; unless stated otherwise, tasks are undiscounted and have terminal states.The ground-truth reward is hidden during learning and used to generate synthetic preferences and evaluate learned policies.Source: §2 · p. 3

2.1 Reward learning from pairwise preferences

  • Reward learning minimizes cross-entropy on pairwise preference labels, while the paper’s focus is how the preference model defines the probability of one segment being preferred over another.Preference samples pair equal-length segments and encode which is preferred, or a tie.Source: §2.1 · pp. 3–4

2.2 Choice of preference model: partial return and regret

  • The partial-return model uses a Boltzmann distribution over segment returns, whereas the proposed regret model uses a Boltzmann distribution over negated regret.Regret measures deviation from optimal decision-making; the paper defines deterministic regret and a form that accounts for stochastic transitions.Source: §2.2 · p. 4
  • For deterministic transitions, optimal segments have zero regret and suboptimal segments have positive regret; in a special case, regret and partial-return preferences coincide.The special case has deterministic transitions, terminal states, and equal starting states.Source: §2.2 · p. 4
  • The models can disagree: the car example favors the left segment under partial return but the optimal right segment under regret, while the paper notes assumptions about human expertise and decision quality.The model assumes a Boltzmann choice rule and that people can identify optimal and suboptimal segments; this may be less true in domains where they have less expertise. In stochastic settings it assumes people prefer optimal decisions even when the best outcome may result from suboptimal decisions.Source: Figure 2 · p. 5, §2.2 · p. 5

3 Theoretical comparison

The section compares preference models by asking whether exhaustive preferences identify a reward function with the same optimal policies. It proves identifiability for regret-based preferences and identifies several settings where partial-return preferences do not guarantee it.

3 Theoretical comparison

  • Identifiability asks whether exhaustive preferences recover a reward function with the same optimal policies, independent of the learning algorithm.The analysis treats preference models as generators of preferences and compares their information about the ground-truth reward.Source: §3 · p. 5

3.1 The regret preference model is identifiable.

  • Regret preferences are identifiable for models that prefer lower-regret segments and treat equal-regret segments equally, including stochastic and noiseless preferences.The result includes, but is not limited to, the Boltzmann regret model.Source: §3.1 · pp. 6–7

3.2 The partial return preference model is not generally identifiable.

  • Partial-return preferences can fail to identify optimal policies with noiseless labels, in variable-horizon tasks, and when segments have length 1.The three failures arise from distinct forms of preference insensitivity.Source: §3.2 · p. 7

3.2.1 Partial return is not identifiable when preferences are noiseless.

  • In noiseless settings, different reward functions can induce the same partial-return preferences despite having different optimal actions, including for fixed segment lengths.The examples extend to any fixed segment length; stochastic transitions also yield preferences insensitive to which initial action is optimal.Source: §3.2.1 · pp. 7–8, Figure 3 · p. 7

3.2.2 Partial return is not identifiable in variable-horizon tasks.

  • In variable-horizon tasks, adding a constant to rewards can preserve partial-return preference probabilities while changing optimal policies.For fixed-horizon and continuing tasks, the same constant shift preserves optimal policies; in variable-horizon tasks it can reverse incentives, such as making an agent avoid a goal.Source: §3.2.2 · p. 9

3.2.3 Partial return is not identifiable for segment lengths of 1.

  • With one-transition segments, partial-return preferences reduce to immediate-reward preferences, so they are unaffected by the discount factor and cannot identify it.Because different discount factors can yield different optimal policies, the learning problem can be underspecified; regret preferences remain identifiable when the discount factor is unknown.Source: §3.2.3 · p. 10

4 Creating a human-labeled preference dataset

The paper gathers human preferences over trajectory segments in a delivery task designed to make bad behavior easy to spot while making optimal behavior difficult to determine. Subjects learn the task, compare segment pairs, and complete comprehension checks before their responses are retained.

4.1 The general delivery domain

  • The delivery domain makes bad behavior identifiable while making optimal behavior difficult to determine from most states.The gridworld includes road-surface costs, coins, roadblocks, a destination, and sheep; transitions can end at the destination for +50 or in failure at a sheep for −50.Source: §4.1 · p. 10

4.1.1 The delivery task

  • The gathered dataset uses a 10×10 delivery MDP where reaching the goal has the highest possible return from every state.The start state is uniformly random over non-terminal states in the general domain.Source: §4.1 · p. 10, §4.1.1 · p. 11

4.2 The subject interface and survey

  • Subjects learn the domain through instructions, financial-outcome framing, and interactive episodes before controlling the specific delivery task.They then choose between 40–50 randomly ordered segment pairs, with options to prefer either segment or mark them the same; “can’t tell” responses are excluded.Source: §4.2 · p. 11
  • After comprehension checks, the filtered dataset contains 1812 preferences from 50 subjects.Data were removed for poor quiz performance or preferring sheep collisions over avoiding them.Source: §4.2 · p. 11

4.3 Selection of segment pairs for labeling

  • The second-stage segment pairs were collected specifically to improve the partial return model, and both stages were combined for analysis.Without second-stage data, that model performed worse than a uniformly random policy.Source: §4.3 · p. 12

5 Descriptive results

Human preferences contain information about segments’ state-value changes beyond their partial returns. The regret model also predicts the collected preferences better than the partial return model.

5.1 Correlations between preferences and segment statistics

  • State-value changes influence human preferences independently of partial return, supporting the hypothesis that people consider segment start and end states.For partial-return differences of −1 and −2, rank correlations were significant (r >=0.3, p<0.0001).Source: §5.1 · pp. 12–13

5.2 Likelihood of human preferences under different preference models

  • The regret model has lower cross-entropy loss than the partial return model when predicting the collected human preferences.The analysis uses 10-fold cross-validation to learn a reward scaling factor for each model.Source: §5.2 · p. 13

6 Results from learning reward functions

The paper evaluates regret-based reward learning against partial-return reward learning, using an approximation of regret based on successor features. Across synthetic and human preferences, the regret model tends to yield better-performing learned policies, especially with smaller human-preference datasets.

6 Results from learning reward functions

  • They learn a reward function, use value iteration to find an approximately optimal Q function, and evaluate the maximum-entropy optimal policy’s mean return under the ground-truth reward over D0.That policy chooses uniformly randomly among optimal actions; optimal policies have normalized mean return 1, and scores above 0.9 are considered near optimal.Source: §6 · p. 13

6.1 An algorithm to learn reward functions with regret(σ|ˆr)

  • The regret-learning algorithm uses successor features to approximate optimal state and state-action values for arbitrary reward functions.This approximation supports computing regret during reward learning; the learned reward function is not restricted to the input reward-function set used to create the approximation.Source: §6.1 · pp. 13–14

6.2 Results from synthetic preferences

  • Across 100 deterministic MDPs, regret outperforms partial return at every dataset size, with and without noise.Wilcoxon tests on normalized mean returns found p<0.05 for 86% and p<0.01 for 57% of comparisons; Figure 10 presents the performance comparison.Source: §6.2 · p. 15, Figure 10 · p. 15

6.3 Results from human preferences

  • With human preferences, regret reaches near-optimal performance more often on smaller datasets and almost always has higher mean return, while both models reach near-optimal return with sufficiently large datasets.The regret model’s mean return is higher for all larger-data partitions except the 10-partition test with only 3 partitions; Figure 11 presents performance across preference-set sizes.Source: §6.3 · p. 16

7 Conclusion

The paper concludes that regret-based preference models improve on partial-return models in theory and in evaluations with synthetic and human preferences. It also finds that the choice of preference model affects learned reward functions and policies.

7 Conclusion

  • With infinite and exhaustive preferences, regret identifies the set of optimal policies, while partial return is not guaranteed to do so in multiple common contexts.The paper also reports that regret tends to outperform partial return on synthetic and human preferences.Source: §7 · p. 16
  • Changes in segment state values provide information about human preferences beyond partial return, independently of the proposed regret model.The paper identifies preference-model choice as impactful for learned reward-function performance.Source: §7 · p. 16
  • The paper identifies further testing of regret’s general superiority and application to deep-learning settings as directions for future work.It also proposes developing methods to nudge people toward normatively appealing preference models and seeking models more effective with actual human preferences.Source: §7 · p. 16

A.1 Limitations

The appendix identifies practical and ethical boundaries for applying regret-based preference learning, including computational and human-modeling assumptions, stakeholder representation, and unresolved scaling challenges.

A.1 Limitations

  • The regret approximation can be difficult to construct in complex domains, runs slower than the partial-return model, and may require extra care to avoid overfitting.The computational cost comes from successor-feature computation and softmax approximations; optimization loss was unstable, and more complex reward features may warrant held-out preference data.Source: §A.1 · p. 20
  • The analysis focuses on simple tasks and leaves the interaction between preference models and discount factors for further study.The authors generally assume matching discount factors, while deep RL may require artificially lower factors than the task's true discount factor.Source: §A.1 · p. 20
  • The work does not choose which segment pairs to label, though the authors suggest its algorithm may be combined with active-learning methods.Those methods address preference-query selection and could potentially combine with Pregret's sample efficiency.Source: §A.1 · p. 20
  • The paper does not model training or interface effects, differing stakeholder preferences, or humans' deviations from simple preference models.The authors call for more fine-grained characterization of human preferences to improve alignment of learned rewards.Source: §A.1 · p. 20

A.2 Ethical statement

  • The paper assumes preferences from different people can be combined, without addressing whose interests an autonomous system should prioritize.The authors urge adequate representation of vulnerable and marginalized communities in high-impact settings and note that corporate-only alignment could enable exploitation or harm.Source: §A.2 · p. 21
  • The authors lack demographic data to assess whether their Mechanical Turk subjects represent any specific population.The study's worker filter is described in Appendix D.Source: §A.2 · p. 21

A.3 On the challenge of using regret preference models in practice

  • Regret preferences are more effective when measured precisely or approximated effectively, but scaling those approximations remains an open challenge.The authors point to inverse reinforcement learning's inner-loop computational demands as a related challenge and are optimistic that scaling methods can be adapted.Source: §A.3 · p. 21

B Preference models for learning reward functions

  • Appendix B extends the preference-model and learning-algorithm discussion, including logistic derivation, regret extensions, policy learning, and connections to inverse reinforcement learning.The appendix section is presented as a supplement to Section 2.Source: B Preference models for learning… · p. 21

B.1 Derivation of the logistic expression of the Boltzmann distribution

The appendix derives and extends the paper's preference models, including discounted regret and variants that interpolate between or modify the core models. It also relates regret preferences to inverse reinforcement learning and notes that discounting remains insufficiently studied.

B.1 Derivation of the logistic expression of the Boltzmann distribution

  • The appendix derives the logistic preference expression from a Boltzmann representation by applying the same segment function to each of two segments.The resulting probability is logistic(f(σ1)−f(σ2)).Source: §B.1 · p. 21

B.2 Learning reward functions from preferences, with discounting

  • The discounting extension generalizes partial return and regret expressions, and regret reduces to partial return for deterministic segments sharing a start state and ending terminally.The appendix says its discounting treatment is beyond scope as a rigorous account, leaving its guidance speculative.Source: §B.2 · pp. 22–23

B.3 Logistic-linear preference model

  • The logistic-linear model generalizes partial return and deterministic regret, recovering them with respective weight vectors ⟨0,1,0⟩ and ⟨−1,1,1⟩.The weights can be fitted to maximize likelihood on human preference data.Source: §B.3 · pp. 23–24

B.4 Adding a constant probability of uniformly distributed preference

  • A variant adds a fitted constant probability of uniformly random preference to an existing preference model, allocating half of that probability to each segment.The adaptation differs from Christiano et al. (2017) by fitting the constant from training data in cross-validation.Source: §B.4 · p. 24

B.5 Expected return preference model

  • The expected-return model includes a segment's partial return and end-state value while ignoring start-state value, and regret learning matches or outperforms it.The authors include it as a plausible model of human preferences.Source: §B.5 · p. 24

B.6 Relationship to inverse reinforcement learning

  • A noiselessly optimal demonstrated segment corresponds under regret preferences to preferring it or being indifferent to every other segment, unlike partial-return preferences.The connection does not resolve inverse reinforcement learning's identifiability issues, including uncertainty about which actions are optimal.Source: §B.6 · pp. 24–25

C Theoretical comparisons

The theoretical analysis compares how much reward information noiseless and stochastic preferences provide. Regret-based preferences avoid an identifiability failure that partial-return preferences can have, while the analysis also explains why noiseless cases matter despite stochastic preference models.

C Theoretical comparisons

  • Structured noise can help both models, but partial-return signal may still be insufficient; the authors suggest regret’s stronger use of signal and noise may explain its sample efficiency.The noiseless setting also helps assess sample efficiency under low noise.Source: C Theoretical comparisons · p. 25
  • In some MDPs, noiseless partial-return preferences cannot identify an equivalent reward function under any preference model, whereas regret preferences avoid this issue in noiseless and certain stochastic settings.The partial-return mapping can be many-to-one across reward functions with different optimal policies, so the information needed to recover an equivalent reward function is absent.Source: C Theoretical comparisons · p. 25
  • Noiseless analysis remains relevant because noiseless preferences are feasible, though rare, and noise-dependent identifiability can rely heavily on a specific Boltzmann-rational structure.The passage cites Skalse et al. (2022) on that concern.Source: C Theoretical comparisons · p. 25
  • Prior work also studies noiseless human input, treating apprenticeship-learning demonstrations as noiselessly optimal and examining reward identifiability from noiseless optimal demonstrations.The cited examples are Abbeel & Ng (2004) and Kim et al. (2021).Source: C Theoretical comparisons · p. 25

D.1 The preference elicitation interface and study overview

  • An experimental error prevented the IRB-approved consent form from appearing after workers accepted the study; the IRB approved use of the resulting data.The accompanying survey awarded 1 point for fully correct answers and 0.5 for partially correct answers, discarding participants below 4.5 points.Source: §D.1 · p. 26, Table 2 · p. 26

D.2 Filtering subject data

The study filtered participants using eligibility criteria, a task-comprehension survey, and a rule against preferring sheep collisions. The final dataset contains preferences from 50 subjects.

D.2 Filtering subject data

  • Eligible Mechanical Turk participants had to be in the United States, hold at least a 99% approval rating, and have completed at least 100 other HITs.The criteria were intended to improve the likelihood of attentive participation and understanding of the training protocol.Source: §D.2 · pp. 26–27
  • Participants scoring below 4.5 on the 0–7 task-comprehension survey were excluded; the threshold balanced comprehension standards against retaining enough data.Fully correct answers earned 1 point and partially correct answers earned 0.5.Source: §D.2 · p. 27
  • Data were also removed if a participant ever preferred colliding the vehicle with a sheep over avoiding the collision, which the authors treated as poor understanding or inattention.The authors considered sheep collisions highly undesirable in this task.Source: §D.2 · p. 27
  • After filtering, 50 subjects contributed 1,812 preferences over 1,245 unique segment pairs to the experiments.The study initially collected data from 143 subjects; 58 were removed for survey responses and another 35 for sheep-collision preference errors.Source: §D.2 · p. 27

D.3 The two stages of data collection

The two-stage collection sampled segment pairs to characterize human preferences and to test differences between regret and partial-return models. The second stage added pair types intended to address weaknesses observed in the first-stage reward-learning evaluation.

D.3 The two stages of data collection

  • The first-stage sample deliberately covered a range of state-value and partial-return differences, with extra sampling where regret and partial-return models disagreed.Pairs were drawn from points in Figure 8, then sampled across four segment-pair bins; this stage contributed 1,437 pairs to the 1,812-pair reward-learning dataset.Source: §D.3 · pp. 27–28, Figure 12 · p. 27
  • With first-stage data alone, the partial-return model performed worse than uniformly random, while the regret model was near-optimal; the second stage was designed to better assess partial return.The authors speculated that terminal/non-terminal pairs and pairs ending at different times could help distinguish terminal rewards and per-move gas costs.Source: §D.3 · p. 28
  • The second stage sampled additional terminal/non-terminal and shifted-terminal pairs, presenting 50 randomly selected pairs per subject when available.The terminal/non-terminal pairing used each segment once; shifted pairs compared original terminal segments with versions moved earlier and padded with absorbing-state transitions.Source: §D.3 · pp. 28–29, Figure 13 · p. 28

D.4 The study design pattern

  • The study design taught subjects a task metric or incentivized alignment with it, collected their input for an algorithm unaware of that metric, and evaluated outputs against the hidden metric.The passage identifies Cui et al. (2020) as another example of this design pattern.Source: §D.4 · p. 29

E Descriptive results

  • The appendix derivation is stated to support an assertion in Section 5.1, but the supplied passage does not specify that assertion or its derivation.Source: E Descriptive results · p. 29

E.2 Losses of an expanded set of preference models on the human preferences dataset

The expanded human-preference analysis compares additional preference models and reports learned weights for a logistic linear model. The accompanying reward-learning material describes implementation choices, policy inputs for successor-feature learning, and possible reward features when linearity is unsuitable.

E.2 Losses of an expanded set of preference models on the human preferences dataset

  • The logistic linear preference model provides a lower bound because it can express either the regret or partial-return model, and its three parameters showed no observed overfitting.Table 3 reports mean cross-entropy test loss over 10-fold cross-validation on 1,812 preferences; the supplied passages do not provide the table’s cell values.Source: §E.2 · p. 29, Table 3 · p. 29
  • Adding a constant probability of a uniformly random response produced no observed likelihood benefit, so the authors excluded that augmentation from further analysis.The comparison used the best test loss across runs with three initializations of the model’s logistic parameter.Source: §E.2 · p. 29
  • The learned logistic-linear weights were closer to the regret model’s weights than to the partial-return model’s, with start-state value weighted less than in the regret model.The scaled weights were ⟨−0.53, 1.0, 0.94⟩ for start-state value, partial return, and end-state value, respectively.Source: §E.2 · p. 29

F.1 An algorithm to learn reward functions with regret(σ|ˆr)

  • The regret model’s cyclical training-loss fluctuations led the authors to select the reward function with the lowest loss across epochs; with more complex reward representations, they say early stopping likely should use validation loss.They found no evidence of overfitting with their linear reward representation and note that understanding the fluctuations could further improve regret-based learning.Source: §F.1 · p. 30

Reward learning with the partial return preference model; Reward learning with the regret preference model

F.1.1 For Algorithm 1, choosing an input set of policies for learning successor feature functions

  • Algorithm 1 uses input policies to build successor-feature functions, then applies generalized policy improvement to estimate optimal value functions for the learned reward during training.The authors’ intuition is that diverse successor features and some reasonably performing input policies can improve generalized policy improvement.Source: §F.1.1 · p. 31
  • The policy-input set may come from randomly sampled reward functions, structured reward-function grids, imitation policies, or iterative bootstrapping from the current learned reward.The article’s own procedure samples reward functions; the other options are proposed alternatives.Source: §F.1.1 · p. 31

F.1.2 Instantiating Algorithm 1 for reward functions that may be non-linear

  • Algorithm 1 assumes a linear reward-feature representation, while one-hot features for every discrete transition lack generalization across examples and related tasks.When features are unknown or rewards are nonlinear, the authors propose features that permit a linear approximation, including learned features from self-supervised auxiliary tasks or a partial-return reward model.Source: §F.1.2 · p. 32

F.2.1 Learning reward functions from 100 randomly generated MDPs

Across randomly generated MDPs, the regret preference model generally outperforms partial return, while the partial-return model’s identifiability and performance depend on handling variable-length trajectories. The proposed absorbing-state adjustment helps address one source of ambiguity but assumes zero reward after termination.

F.2.1 Learning reward functions from 100 randomly generated MDPs

  • The regret model outperforms partial return in most conditions when performance is measured against a uniformly random policy, and achieves near-optimal performance on a higher proportion of MDPs.Figure 14 uses the random-policy threshold; Figure 10 reports near-optimal performance.Source: §F.2.1 · p. 33, Figure 14 · p. 33

F.2.2 The effect of including transitions from absorbing state

  • In variable-horizon tasks, reward functions differing by a constant can yield identical partial-return preference probabilities yet different optimal policies.The suggested adjustment forces one reward output to zero and includes a segment containing that transition; the authors describe this as an imperfect fix.Source: §F.2.2 · pp. 33–34
  • The adjustment anchors reward inference by setting transitions from the absorbing state to zero and including early-terminating segments in practice.An absorbing state is the theoretical state entered when the task terminates; all actions there loop back to that state.Source: §F.2.2 · p. 34
  • The absorbing-state adjustment assumes zero reward after termination, which may misrepresent preferences when termination lets an agent pursue other positively rewarded tasks.The authors also note that fixing any reward to zero adds task knowledge beyond the preference data and may not align with the ground-truth reward.Source: §F.2.2 · p. 34
  • Without early-terminating segments, partial-return performance worsens, while regret performance appears insensitive to their inclusion.The comparison uses new randomly generated MDPs and learning rates selected on a separate set of 30 MDPs.Source: Figure 15 · p. 34, §F.2.2 · p. 35

F.2.3 Varying segment length

Segment length affects the two preference models differently, and regret remains strong in stochastic MDPs and across several human-preference analyses. The results also show that performance depends on whether the preference generator matches the model used for reward learning.

F.2.3 Varying segment length

  • With noiseless preferences, regret performs better or similarly for segment lengths below 20; with stochastic preferences, it generally performs better and partial return has higher variance.At length 20, noiseless results are similar, sometimes slightly favoring partial return; stochastic results sometimes marginally favor partial return.Source: §F.2.3 · p. 36, Figure 18 · p. 36, Figure 19 · p. 36
  • Increasing segment length improves partial-return performance in noiseless and stochastic conditions, while regret shows no significant effect noiselessly and only a minor improvement stochastically.Reported mean Kendall’s τ values are 0.80 and 0.51 for partial return, and 0.25 and 0.47 for regret, respectively.Source: §F.2.3 · pp. 36–37

F.2.4 Reward learning in stochastic MDPs

  • In stochastic MDPs with varied reward functions, Pregret achieved near-optimal performance in all 10 MDPs for both noiseless and stochastic preference datasets, whereas PΣr did not.Source: §F.2.4 · p. 37, Table 6 · p. 37

F.2.5 Generating preferences and learning reward functions with different preference models

  • Each preference model performs best when learning from labels it generated; regret learning on regret-generated preferences is the best-performing combination.Among the mixed-model conditions, partial-return learning on regret-generated preferences outperforms regret learning on partial-return-generated preferences.Source: §F.2.5 · p. 38, Figure 20 · p. 38

F.3.1 Wilcoxon paired signed-rank test

  • On human preferences, regret learning tends to yield higher mean scores than partial return, even when both models reach near-optimal performance.Both models reached 100% near-optimal performance for five and ten partitions, yet paired tests showed significant differences.Source: §F.3.1 · p. 38, Table 7 · p. 38

F.3.2 Expanded plots, with the expected return and logistic linear preference models included

  • Across every partitioning of the human data, regret matches or outperforms the expected-return and logistic-linear models at both near-optimal and better-than-random thresholds.Figure 21 adds the expected-return and logistic-linear models; Figure 22 uses the better-than-random threshold.Source: §F.3.2 · p. 38

F.3.3 Performance on only human preferences from the first stage of data collection

On the first-stage human-preference data, regret achieves near-optimal performance while partial return performs worse than random. Removing early-terminating segments reproduces that poor partial-return result, while generalization to other MDPs shows a less decisive advantage.

F.3.3 Performance on only human preferences from the first stage of data collection

  • With 1,437 first-stage human preferences, partial return achieves normalized mean return −12.7, below random, while regret reaches 0.999 against an optimum of 1.0.The first-stage dataset contains 79% of the full preference dataset.Source: §F.3.3 · p. 38, Figure 21 · p. 39

F.3.4 Performance without early-terminating segments

  • Removing only early-terminating segment pairs reproduces the first-stage results: −12.7 for partial return and 0.999 for regret.The authors conclude that removing these segments is sufficient to cause poor partial-return performance.Source: §F.3.4 · p. 39

F.3.5 Generalization of learned reward functions to other MDPs with the same ground-truth reward function

  • On other MDPs with the same ground-truth reward, regret performs similarly at four partition counts and better at the others; the largest gaps occur at 20 and 5 partitions.At those two partition counts, partial return fails to reach near-optimal performance approximately twice as often as regret.Source: §F.3.5 · pp. 39–40, Figure 23 · p. 39
  • Across all seven training sizes, paired tests find no statistically significant generalization advantage for regret (p > 0.2).The authors therefore cannot conclude superior performance from assuming the regret preference model in this analysis.Source: §F.3.5 · p. 40

Related papers

Loading 2206.02231v3…