Causal Confusion and Reward Misidentification in Preference-Based Reward Learning

1.1 RELATED WORK

• Prior work examined proxy-reward misspecification, behavioral-cloning confusion, and goal misgeneralization; this study systematically examines confusion while learning rewards from preferences.Source: §1 · p. 1, §1.1 · p. 2

1 INTRODUCTION

• In assistive feeding, a learned signed spoon–mouth difference can favor moving behind the patient’s head over feeding them.The signed difference correlates with absolute distance in training data, but does not represent the intended distance-based behavior.Source: §1 · p. 1
• Even when preference data are abundant, learned rewards can have high test accuracy yet produce poor policies when optimized.The study examines non-causal distractors, model capacity, preference noise, and partial observability, and highlights data coverage and interactive feedback as relevant directions.Source: §1 · pp. 1–2

1.2 BACKGROUND: REWARD LEARNING FROM PREFERENCES

• The setup learns a reward from preference labels over trajectory pairs; Figure 1 distinguishes nuisance variables, unobserved causal features, and user noise.Source: §1.2 · p. 3, Figure 1 · p. 3

• Preference learning may include causal features x, correlated nuisance features z, unobserved causal features u, and unobserved human bias or noise η.Source: §2 · p. 3
• Reward misidentification is a learned reward with low test error that produces poor true-reward performance when optimized via RL.Source: §2 · p. 3
• A learned reward can rely on nuisance features correlated with causal features or on faulty correlations that hold only in the training distribution.Source: §2 · p. 3
• Optimizing a misidentified reward can shift behavior out of distribution and yield low performance under the true reward; the causal diagrams illustrate typical, not exhaustive, sources.Source: §2 · p. 3

• The benchmarks are Reacher, Feeding, and Itch Scratching, covering target reaching, feeding a person, and scratching an itch location.Source: §3 · p. 4
• Synthetic demonstrations and preference labels come from noise-injected expert policies trained on predefined true rewards, which are withheld during policy learning.The learned reward is trained from preferences, then used for RL; policies are evaluated on the true reward.Source: §3 · p. 4
• Figure 2 contrasts high test accuracy in the left two columns with poor learned-reward behaviors in the rightmost column.Predicted trajectory rewards appear beneath each image.Source: Figure 2 · p. 4
• Evaluation checks held-out test error, true-reward policy performance, and whether the learned reward prefers its optimized policy over the true-reward-optimal policy.The study also uses saliency maps, EPIC distance, and KL divergence to analyze reward errors and induced distribution shift.Source: §3 · p. 4

4 EVIDENCE OF CAUSAL CONFUSION

• In this generous setting, added features expose all information needed for TRUE, and learned rewards achieve high preference test accuracy comparable to training.The setup also uses large amounts of preference data.Source: §4 · p. 4
• Despite high pairwise preference test accuracy, PREF policies perform very poorly under TRUE versus GT, while the learned reward consistently prefers PREF.The table compares preference-trained and true-reward policies optimized with 1M RL iterations and averaged over 3 seeds.Source: Table 1 · p. 5, §4 · p. 5
• Reacher spins instead of reaching, Feeding minimizes signed spoon–mouth difference, and Itch Scratching circles the arm while keeping its end effector near the itch target.For Itch Scratching, the learned reward spuriously increases weights on two action components; for Feeding, front-of-face approaches correlate signed difference with distance.Source: §4 · p. 5

5.1 DISTRACTOR FEATURES

• Removing distractors drastically improves Reacher performance, while each retained distractor worsens it almost linearly and increases distribution shift.Positive gradients on high-range velocity features let spinning exploit a slight training-data bias, moving RL into states not seen during reward learning.Source: §5.1 · pp. 6–7, Figure 3 · p. 6

5.2 MODEL CAPACITY

• Increasing reward-model capacity does not necessarily improve the performance of the policy optimized against it.The analysis reports this despite careful hyperparameter tuning across model and dataset sizes.Source: §5.2 · p. 7

5.3 NOISE IN STATED PREFERENCES

• Preference noise can leave test accuracy high while policy performance plunges; Stochastic and Skip accuracy can match or exceed Oracle despite much worse performance.Validation accuracy also fails to reliably predict alignment: a model at 0.995 can align worse than one at 0.985.Source: §5.3 · p. 7
• Noisy rewards can mimic ground truth on the reward-learning distribution yet fail under RL, where their learned state-action distribution diverges more.EPIC comparisons reverse depending on whether coverage is drawn from reward learning or RL; the KL divergence is greater for Stochastic noise.Source: §5.3 · p. 7

5.4 PARTIAL OBSERVABILITY OF CAUSAL FEATURES

• Reward learning succeeds under partial observability only when all causal reward features are observable, although ground-truth-reward RL can still learn proper feeding.Without food observability, a learned policy can bring the spoon near the mouth but spill the food rather than deliver it.Source: §5.4 · pp. 7–8, Figure 6 · p. 8
• Partial observability makes the reward model overweight available causal features and exploit non-causal correlations, including joint angles and action components irrelevant to the true reward.The resulting behavior prioritizes spoon-mouth distance without ensuring food stays on the spoon or reaches the mouth.Source: §5.4 · p. 8

5.5 COMPLEX CAUSAL FEATURES

• Preference-based reward learning fails to produce a successful itch-scratching policy even with more data, greater capacity, no distractors, perfect preferences, and full observability.Model-free RL using the ground-truth reward solves the task.Source: §5.5 · p. 8
• Even with low-level information for scratching, the learned reward misses the motion; adding a high-level “scratched” feature sharply improves performance.The authors suspect that consistent spurious correlations involving a few variables impede learning a causal relation spanning many variables that matter only in particular contexts.Source: §5.5 · pp. 8–9, Figure 7 · p. 9

6 CONCLUSION

• Optimizing misidentified rewards can push policies outside the training distribution, where learned rewards are high but true rewards are poor.The authors examine this shift using saliency maps, EPIC distance, KL divergence, and resulting behaviors.Source: §6 · p. 9
• Non-causal distractor features and noisy preferences aggravate reward misidentification, while partial causal-feature observability leaves reward models struggling.The authors also report that a high-performing policy can exist using only observable state information.Source: §6 · p. 9
• A direct KL-divergence penalty during RL did not stop reward hacking and made performance worse in the tested setup.The authors attribute this to an additional degree of freedom in the reward function, while hypothesizing other penalty approaches may work.Source: §6 · p. 9
• Reward models can appear close to the true reward on training data yet fail when optimized, motivating further work on distribution penalties, feature feedback, and high-level feature learning.The authors also identify alternative reward comparisons and active, iterative data acquisition as possible directions.Source: §6 · p. 9

A.1 PREFERENCE LEARNING TRAINING DETAILS

• Given pairwise preference labels over trajectories, the procedure optimizes a parameterized reward function to maximize their likelihood.The reward maps state-action pairs to scalar rewards.Source: §A.1 · p. 13
• The reward likelihood is differentiable; the authors use Adam for reward learning and PPO or SAC for policy optimization.PPO is used for Feeding and Itch Scratching, while SAC is used for Reacher.Source: §A.1 · p. 13
• Training uses 780, 7,140, or 52,326 unique comparisons, with a 1,770-preference test set drawn from 60 disjoint trajectories.Learning rate and weight decay are coarsely tuned using the medium dataset size.Source: §A.1 · p. 13
• Reward models use two hidden layers of 128 and 64 units with Leaky ReLU, Adam weight decay, validation-loss early stopping, and 1,000,000 policy-optimization timesteps.Early stopping uses a patience of 10 epochs; the policy optimizer is PPO or SAC depending on the environment.Source: §A.1 · p. 13
• The algorithm computes reward estimates for preferred trajectory pairs, applies cross-entropy loss, validates for early stopping, then runs RL using the learned reward.The validation loss stopping patience is 10 epochs.Source: §A.1 · p. 14

A.2 SYNTHETIC PREFERENCE GENERATION

• An expert policy trained on each environment’s ground-truth reward generates diverse trajectories by adding ε-greedy action noise.ε = 0 gives the trained policy, while ε = 1 gives a uniformly random policy; the cited work notes expected performance decreases monotonically with this disturbance (Brown et al., 2020b).Source: §A.2 · p. 14
• The datasets use 40, 120, or 324 trajectories, sampled from rollouts at ε values of 0, 0.2, 0.4, 0.6, 0.8, and 1.These trajectory sets correspond to the small, medium, and large datasets, respectively.Source: §A.2 · p. 14
• All trajectory pairs are labeled using ground-truth rewards, with held-out trajectory sets reserved for validation and testing.Source: §A.2 · p. 14

A.3 EVIDENCE OF CAUSAL CONFUSION

• In Half Cheetah, the learned reward is misidentified at the small size but aligns more closely with the true reward at medium and large sizes.At the large size, poor policy performance is attributed more to RL failure; the authors note the true reward depends essentially on observed x-velocity.Source: §A.3 · p. 14, Table 3 · p. 16
• Feature-correlation plots and gradient saliency maps expose improper reward weighting, including positive ‘diff y’ gradients that incentivize moving behind the human’s head.A properly learned reward would turn that gradient negative once the robot arm passes behind the head.Source: §A.3 · p. 14, Figure 9 · p. 15, Figure 8 · p. 15
• Across three tasks and all dataset sizes, PREF policies perform poorly under the true reward despite high pairwise accuracy, while learned rewards consistently prefer PREF over GT.The table reports 1M RL iterations averaged over three seeds; task features are fully observable.Source: Table 2 · p. 16

• Input features are labeled causal (C) or distractor (D), with each feature’s dimensions shown.Source: §A.4 · p. 16
• The listed Lunar Lander inputs include position coordinates, linear velocity, lander angle, leg contact, and engine-action choices.Source: §A.4 · p. 16
• Distractor examples include arm angles and angular velocities, robot joint angles, and absolute spoon or target positions and orientations.Source: §A.4 · p. 17
• Listed items include fingertip position relative to target (3), torque actions at first and second hinges (2), spoon position relative to target (3), spoon force on human (1), actions (7), and foods in mouth (1).Source: §A.4 · p. 17

A.5 GROUND TRUTH REWARD FUNCTIONS FOR ENVIRONMENTS; A.6 EVALUATING LEARNED REWARD FUNCTIONS

• Ground-truth rewards combine task-specific terms, while learned rewards are examined using saliency maps, EPIC comparisons, and estimated distribution shift.Source: §A.5 · p. 18, §A.6 · pp. 18–19

A.7 DISTRACTOR FEATURES

• Removing distractors generally improves performance, but does not resolve Feeding’s spurious correlation because it involves a causal feature: spoon-to-target position difference.Source: §A.7 · p. 19, Figure 10 · p. 20

A.8 MODEL CAPACITY

• Greater reward-model capacity does not reliably improve policy performance and can worsen generalization from reward-learning states to reinforcement-learning states.Source: §A.8 · pp. 19, 21, Table 5 · p. 20

A.9 NOISE IN STATED PREFERENCES

• With noisy preferences, more data can reduce performance, and combining noise with distractors produces a larger performance loss.Source: §A.9 · pp. 21–22

A.10 PARTIAL OBSERVABILITY OF CAUSAL FEATURES

• Partial observability can cause over-weighting of available causal features and spurious correlations, so the Feeding robot approaches the mouth without ensuring food reaches it.Source: §A.10 · p. 23, Table 11 · p. 25

A.11 COMPLEX CAUSAL FEATURES

• For the multi-variable scratching behavior, adding a hand-specified “scratched” indicator drastically increases performance; without it, the learned reward induces greater distribution shift.Source: §A.11 · p. 25, Table 12 · p. 25

A.12 DATA GENERATION METHODS; A.13 PENALIZING THE KL DIVERGENCE

• Diverse preference trajectories perform on par with or better than checkpointed RL rollouts, while an offline KL penalty does not alleviate misidentification and can worsen performance.Source: §A.13 · p. 25, Figure 15 · p. 26

A.14 ITERATIVE PREFERENCE-BASED REWARD LEARNING FROM ONLINE DATA

• With online preference queries added iteratively, test error remains very low while policy performance measured by true reward and success remains poor.Source: §A.14 · p. 27