Inducing Structure in Reward Learning by Learning Features

• Hand-crafted features are hard to specify exhaustively, while deep IRL needs extensive, diverse input to disambiguate reward functions and generalize.Task demonstrations teach the reward rather than the features themselves, so the robot may find multiple explanations for the same behavior.Source: §1 · p. 1
• Feature traces teach complex features separately from the reward; FERL then uses demonstrations or other reward-learning input to combine them.A feature trace is a partial trajectory that shows monotonic change in the value of the feature being taught.Source: §1 · p. 2
• In the online setting, learning a missing feature yields a more generalizable reward than training deep IRL directly from raw state and known features; offline FERL outperforms its baseline most of the time.Results are less clear when novice users teach noisy features in simulation.Source: §1 · p. 2
• The work extends Bobu et al. (2021) with a formal feature-based framework and an offline setting that teaches features one at a time before their combined reward.The authors note that offline reward learning is more commonly encountered and that the approach applies to different kinds of human input.Source: §1 · p. 2
• FERL improves reward-learning sample complexity and generalization by focusing human input on learning features before the reward.The approach is demonstrated with a physical 7-DoF robot arm and in a simulated user study.Source: §1 · p. 2

2 Related Work

• Imitation learning copies demonstrated behavior without learning why it is desirable, whereas reward learning infers a reward; human input can include demonstrations, corrections, comparisons, rankings, goals, or proxy objectives.Reward optimization can generalize to unseen situations, unlike imitation learning, which works only in the training regime.Source: §2 · pp. 2–3

2.1 Feature Representations in Reward Learning

• Hand-crafted features can provide useful inductive bias but are difficult to design and may omit preferences; deep IRL shifts feature extraction to raw-state function approximators, requiring diverse input to generalize.With expressive models, many rewards can explain the input, and insufficient disambiguation can lead to unintended behavior away from demonstrations.Source: §2.1 · p. 3

2.2 Learning Features

• Earlier feature-learning approaches use demonstrations, dimensionality reduction, logical conjunctions, or regression trees, but retain constraints such as engineered base features or limited state-space suitability.FERL instead learns complex nonlinear features from explicit human input, then infers a reward with minimal input.Source: §2.2 · p. 3

• The robot and human affect a continuous state through their actions, and the robot seeks a parameterized reward that captures the human’s consistent preference ordering over trajectories.The robot does not observe the true reward and estimates its preferred parameters from the human’s actions.Source: §3 · pp. 3–4
• A finite family of features represents the reward, and restricting the representation to those features truncates the full reward-function space.The chosen feature set affects which rewards can be learned, regardless of whether the reward function is linear or a neural network.Source: §3 · p. 4
• The framework solicits feature-specific human input to add features, then supports offline reward learning from demonstrations or online updates after corrections to an incomplete feature set.Source: §3 · p. 4

4.1 Feature Traces

• Feature traces are monotonically decreasing state sequences that provide many state comparisons while relaxing the need for precise value labels.The person guides the system from a start state to an end state, aiming to preserve monotonicity.Source: §4.1 · p. 4

4.2 Learning a Feature Function

• A neural network maps raw states to nonnegative feature values and is trained to distinguish which of two states has the higher value.The feature representation can capture arbitrarily complex nonlinear mappings.Source: §4.2 · p. 5, §4.2.1 · p. 5

4.2.1 Monotonicity Along Feature Traces.

• Ordered state pairs from each trace train feature values to decrease along its sequence, but this loss alone does not align values across trace starts or ends.Each earlier state is paired with later states to form the ordered training data.Source: §4.2.1 · p. 5

4.2.2 Start/End Feature Value Equivalence.

• An equivalence loss makes both states in each endpoint pair equally likely to have the higher feature value, aligning starts and ends across traces.The total objective combines ordered and equivalence losses, with λ trading them off.Source: §4.2.2 · p. 5

4.2.3 Incorporating Relative Values.

• Optional relative endpoint values let traces begin or end inside the feature’s range, supporting richer feature teaching than fixed highest-to-lowest endpoints.The default endpoint values are 1 and 0; relative values can relax that assumption.Source: §4.2.3 · pp. 5–6

5.1 Offline FERL

• Offline FERL first learns features from traces and then learns reward weights from demonstrations, separating feature teaching from reward learning.The framework can also be applied to reward-learning methods beyond demonstrations and corrections when they use features.Source: §5.1 · p. 6, §5 · p. 6

5.1.1 Creating the Feature Set.

• With an initially empty feature set, the person teaches each relevant feature in turn before the robot combines them using standard learning-from-demonstration procedures.The person can add features sequentially and repeat the process as needed.Source: §5.1.1 · p. 6

5.1.2 Offline Reward Learning.

• Offline reward learning interprets demonstrations as evidence about reward parameters under a noisily optimal human-choice model.The model assigns trajectory probabilities proportional to exponentiated reward; the offline formulation fixes β to 1.Source: §5.1.2 · p. 6

5.2 Online FERL

• Online FERL checks whether the current features explain an intervention; if not, the robot learns a feature and then uses the input to update its reward.The robot starts with an initial feature set and a trajectory optimized under its current reward estimate.Source: §5.2 · p. 7

5.2.1 Online Reward Update.

• For corrections, the reward update moves parameters in the direction of the feature change from the current trajectory to the induced trajectory, after which the robot replans.The implementation updates only the weight for the feature that changes most; the authors use corrections, though demonstrations can also supply the human trajectory.Source: §5.2.1 · pp. 7–8

5.2.2 Confidence Estimation.

• The robot treats β as confidence that its features explain human input and requests a new feature when estimated confidence falls below threshold ϵ.For corrections, confidence is low when the human input differs substantially from an input that would achieve the induced features efficiently; demonstrations can use a Bayesian belief update.Source: §5.2.2 · p. 8

6.1.1 Experimental Design.

• On a 7-DoF JACO arm, six personal-robotics features were learned with neural networks from expert traces; between-objects required explicit endpoint values.The networks had two 64-unit layers; the study collected 20 traces per feature, or 40 for test-laptop-location and between-objects.Source: §6.1.1 · pp. 8–9

6.1.3 Quantitative Analysis.

• Across features, FERL beat random (F(1, 113) = 372.0123, p < .0001); error and standard error decreased as trace counts increased (F(8, 526) = 21.1407, p < .0001; F(8, 45) = 3.098, p = .0072).Figure 4 reports normalized MSE means and standard errors across 10 seeds as trace counts increase.Source: Figure 4 · p. 9, §6.1.3 · p. 10

Dependent Measures.

• Feature error was measured against ground truth on 10,000 reachable test states, many far from training traces, and normalized so random initialization scored 1.0.For each trace count, 10 experiments sampled different trace sets.Source: Dependent Measures · p. 9

6.1.2 Qualitative Results.

• Learned features resembled ground truth for most tasks, but between-objects captured only its broad shape in the full input space; a smaller input space recovered finer structure.The authors attribute the difficulty to spurious correlations and suggest more data or active learning for complex features.Source: §6.1.2 · p. 10

6.1.4 Summary.

• Table and laptop performed well with only N = 4 traces, while between-objects required more; targeted active-learning queries could further reduce the data needed.The authors conclude that relatively few traces can yield features that generalize to unseen states.Source: §6.1.4 · p. 10

6.2 User Study

• A simulated JACO study recruited 12 technically experienced campus users, who each gave 10 traces for table, laptop, and proxemics after familiarization.The authors caution that results speak to FERL usability with this population, not the general population.Source: §6.2.1 · p. 10, Manipulated Variables · p. 10, Participants · p. 11

Objective

• User-taught features’ MSEs were closer to expert than random; Random differed significantly from User and Expert, but Expert–User did not (p = .0964).The input-type effect was significant (F(2, 103) = 132.7505, p < .0001).Source: Figure 7 · p. 11, Objective · p. 11

Subjective.

• Users reported positive protocol experiences, though intuitiveness evidence was weaker and positivity bias was possible; post-visualization, they tended to feel more effective and thought the robot learned the correct feature.Objectively stronger teachers were more confident, felt more successful, and found teaching easier (Q3 p = .001; Q4 p = .0203; Q7 p = .0202).Source: Subjective · p. 11, §6.2.3 · p. 11

7.1.2 Qualitative Comparison.

• With expert features, FERL captures a fine-grained reward resembling ground truth, while ME-IRL does not adequately trade off the active features.The comparison covers Laptop Missing, Table Missing, and Proxemics Missing.Source: Figure 8 · p. 12, §7.1.2 · p. 13
• FERL’s reward MSE is lower than ME-IRL’s across three tasks and training-data amounts, with a significant method effect (F(1, 595) = 335.5253, p < .0001).FERL’s MSE decreases with more data, whereas ME-IRL shows no consistent decrease; ME-IRL also has higher standard error.Source: Figure 9 · p. 12, §7.1.3 · p. 13
• FERL produces trajectories preferred under the ground-truth reward over ME-IRL trajectories, with a significant method effect (F(1, 596) = 14.9816, p = .0001).The reward-ratio benchmark compares induced trajectories against the ground-truth ratio of 1.Source: §7.1.3 · p. 13, Figure 10 · p. 13
• FERL assigns higher likelihood than ME-IRL to test trajectories optimized under the ground-truth reward, supporting greater reward-model accuracy.The likelihood comparison uses the Boltzmann noisily-rational assumption.Source: §7.1.3 · p. 13, Figure 11 · p. 13

7.2.2 Analysis.

• Even with non-expert simulation features, FERL outperforms ME-IRL using physical expert demonstrations, and the method effect is significant (F(1, 62) = 41.2477, p < .0001).The comparison also includes FERL using expert-taught simulation features; the user-feature result supports online reward learning despite feature-quality degradation.Source: §7.2.2 · p. 14, Figure 12 · p. 14
• Non-expert features preserve enough structure to benefit online reward learning, although the features are lower quality than expert-taught ones.The authors suggest non-experts can use online FERL to teach robot preferences.Source: §7.2.2 · p. 14

8.1.2 Qualitative Comparison.

• FERL learns a fine-grained reward resembling ground truth across the three tasks; ME-IRL recovers the simple One Feature reward but not the feature trade-offs in more complex tasks.The visual comparison covers One Feature, Two Features, and Three Features.Source: §8.1.2 · p. 15
• Across tasks and data amounts, FERL is closer to ground truth than ME-IRL, with a significant method effect (F(1, 535) = 148.8431, p < .0001).ME-IRL eventually learns a good reward for One Feature, but underperforms FERL on the more complex multi-feature tasks.Source: §8.1.3 · p. 15, Figure 14 · p. 15
• For multi-feature tasks, FERL produces better trajectories under the ground-truth reward and assigns higher probability to test trajectories than ME-IRL; the methods are comparable on One Feature.The behavior-accuracy method effect is significant for Two Features and Three Features alone (F(1, 397) = 5.7489, p = .0097).Source: §8.1.3 · p. 15, Figure 15 · p. 16, Figure 16 · p. 16

8.2.2 Analysis.

• No significant method effect appears when offline FERL combines noisy novice simulation features into a reward, so those features do not reliably outperform ME-IRL.The authors attribute this possibility to the difficulty of teaching features in simulation and the 2D interface for 3D traces.Source: §8.2.2 · p. 16, Figure 17 · p. 16

8.2.3 Summary.

• The novice-feature comparison is qualified because ME-IRL used all-expert physical data while FERL used novice simulation data; the authors suggest physical demonstrations could make FERL’s benefits more prominent.They caution that very noisy feature structure may diminish the benefits of divide-and-conquer learning.Source: §8.2.3 · p. 16

9 Discussion

• Online FERL outperforms ME-IRL in data-efficiency and generalization; offline FERL beats it with sufficiently good features, but results are less conclusive with very noisy features.These findings come from experiments with a real robot arm and a simulation user study.Source: §9 · p. 17

Implications for Online Reward Learning.

• Online FERL lets people teach a missing feature when the robot detects that its feature set is insufficient, then updates the reward using the expanded set.The experiments use reward corrections; adapting the framework to other online methods requires a way to detect an insufficient feature set.Source: Implications for Online Reward Learning · p. 17
• Demonstrations teach both what matters and how to trade off those factors, while feature traces also provide relative values for states off optimal trajectories.Separating feature learning from reward combination lets the robot use information about high-valued but undesirable states and low-valued states where other features matter more.Source: Implications for Online Reward Learning · p. 17

Limitations and Future Work.

• The user study tested feature teaching in simulation, mostly with technically trained users, and did not test teaching features users know implicitly.How nontechnical users would perform, how kinesthetic traces would affect usability, and whether users can teach implicitly known features remain open questions.Source: Limitations and Future Work · p. 17
• Abstract features such as comfort may be hard to teach, and the current protocol may be cumbersome for discontinuous features such as constraints.The authors suggest active selection of informative starting states or combining feature traces with other structured input.Source: Limitations and Future Work · p. 17
• Reward quality depends directly on feature quality: noisy features still helped online learning with one missing feature, but combining multiple noisy features offline showed no benefit.The authors identify deciding whether to accept or reject a learned feature as future work.Source: Limitations and Future Work · p. 17
• FERL works reliably in 27D, but extending it to higher-dimensional state spaces such as images requires further work.The authors note that spurious correlations can reduce feature quality in large input spaces under low-data conditions.Source: Limitations and Future Work · pp. 17–18

A.1 Incorporating Relative Values in Training

• A default start value of 1 sets the start at the feature maximum, while a default end value of 0 sets the end at the minimum.The stated examples use a relative value of 0.3 to illustrate non-extreme endpoints.Source: §A.1 · p. 20
• Relative start and end values shift feature values in proportion to the feature range before the training procedure compares them.A start value of 0.3 places the start 30% above the minimum; an end value of 0.3 places the end 30% above the minimum.Source: §A.1 · p. 20

B.1 Protocols for Feature Trace Collection

• For laptop features, traces start at the laptop’s center and move away by the desired bump radius, with varied directions, heights, and training locations.The test laptop location is not seen during training.Source: §B.1 · p. 20
• Coffee traces begin with the cup upside-down and move the end-effector toward upright; between-object traces teach both object regions and the intervening space.The between-object protocol includes several trace patterns, including traces spanning the objects and traces moving away from the line between them.Source: §B.1 · p. 20
• For proxemics, the goal is to keep the end-effector away from the human, more so when moving in front of their face and less so when moving on their side.Source: §B.1 · p. 20

B.2 Protocols for Demonstration Collection

• Online demonstrations mix trajectories focused on the unknown feature, the known feature, or both, with diverse start and goal configurations.The appendix reports 20 demonstrations for each unknown feature.Source: §B.2 · p. 20
• For Laptop Missing and Table Missing, demonstrations vary whether the robot must avoid the laptop, stay close to the table, or account for both features.Table Missing includes more demonstrations focused on staying close to the table when the laptop is already far away.Source: §B.2 · p. 20
• Proxemics Missing demonstrations vary how aggressively the robot avoids the person, along with height and start-goal locations, to teach features separately and together.Some trajectories avoid the person slightly to the side; others avoid them more aggressively in front.Source: §B.2 · pp. 20–21
• Figure 19 colors demonstrations by true reward, with yellow indicating low values and blue indicating high values.The examples cover Laptop Missing, Table Missing, and Proxemics Missing.Source: Figure 19 · p. 21
• Offline collection used 20 demonstrations for One Feature, 40 for Two Features, and 60 for Three Features, covering individual features and their combinations.The One Feature trajectories begin far from the table and move close; the Three Features set includes each feature separately, every pair, and the full combination.Source: §B.2 · p. 21

B.3 Raw State Space Dimensionality

• The experiments use a 36D state comprising 27 Euclidean coordinates and nine end-effector rotation-matrix entries; a more comprehensive representation is 97D.The 97D representation includes joint angles, joint and object positions, and rotation-matrix entries for all joints.Source: §B.3 · p. 21
• Redundant information in the 97D input can lead to spurious correlations and poorer feature generalization with little data, motivating the reduced 36D state.The authors note that more diverse and numerous data could address the issue, but would increase human effort.Source: §B.3 · p. 21
• The heuristic trains separate subspace models, compares their FERL loss on held-out traces, then trains the lower-loss model on all traces; it selects the right subspace in about 85% of experiments.Each candidate model trains for 10 epochs on half the traces and is evaluated on the other half.Source: §B.3 · pp. 21–22
• Separate-subspace learning yields lower MSEs than learning from all 36 raw-state dimensions, while subspace selection can worsen feature learning when it chooses the wrong subspace.Without the heuristic, spurious correlations particularly reduce feature quality for proxemics and test laptop location; adding the heuristic increases error-bar variance.Source: §B.3 · p. 22, Figure 20 · p. 22
• When the relevant subspace is unknown, the authors suggest using the heuristic or asking the human, while noting that more reliable solutions are needed.The discussion points to causal-learning methods as a possible future direction.Source: §B.3 · p. 22

B.4 User Study Instructions

• The user-study familiarization phase lasted 10 minutes and presented the feature over 10,000 states sampled from the robot’s reachable set.The description says this step supported participants’ standardized understanding of the feature.Source: §B.4 · p. 22

C Implementation Details

• The appendix reports that no extensive hyperparameter tuning was performed and presents the settings that worked best for each method.The authors also state that the code is available online.Source: C Implementation Details · p. 23

C.1 Feature Learning Training Details

• The feature network uses two 64-unit layers with leaky ReLU hidden activations, softplus output, and epoch-wise output normalization.Training balances the fewer equivalence tuples through fivefold augmentation and a 10× loss weight, then uses Adam for 100 epochs with learning rate and weight decay 0.001 and batch size 32.Source: §C.1 · p. 23

C.2 Online FERL Details

• Online, the robot tracks a TrajOpt path, records correction deviations at 100 Hz, and pauses to learn a feature when its explanation confidence falls below 0.1.After feature learning, it uses the recorded deviations to update reward parameters and replan; if its configuration changed, it returns to the last pre-learning configuration before resuming.Source: §C.2 · p. 23
• With 27D joint-and-object positions, the learned between-objects feature misses fine structure; using 9D end-effector-and-object positions improves its quality.The authors attribute the reduction in feature quality with more dimensions to spurious correlations.Source: Figure 21 · p. 23

C.3 Offline FERL Details

• Offline reward learning uses stochastic gradient descent for 50 iterations, generating one TrajOpt trajectory per demonstration start-goal pair at each iteration.The gradient is estimated from the full batch of demonstration tuples.Source: §C.3 · p. 23

C.4 ME-IRL Training Details

• The reward is modeled by a two-layer, 128-unit ReLU network, with known features added before the softplus output in online experiments.Training uses Adam for 50 iterations with learning rate and weight decay 0.001, generating TrajOpt trajectories for demonstration start-goal pairs.Source: §C.4 · p. 23
• At each iteration, the method generates near-optimal trajectories for the current reward and estimates gradients from demonstration and generated trajectories for a start-goal pair.The batches cover all start-goal pairs over the 50 iterations.Source: §C.4 · p. 23
• Figure 22 shows deep ME-IRL outperforms shallow ME-IRL in MSE to ground-truth reward, comparing shallow models with 1, 2, 3, 5, and 10 random features across one-, two-, and three-feature tasks.Source: Figure 22 · p. 24

D.1 Between Objects with 9D State Space

• Restricting the state to 9D end-effector and object positions improves the learned between-objects feature’s fine-grained structure over the 27D joint-and-object state.The authors attribute the poorer high-dimensional result to spurious correlations.Source: §D.1 · p. 24

D.2 Baseline Comparison

• Deep ME-IRL consistently outperforms shallow ME-IRL with 1, 2, 3, 5, or 10 random features across the three tested feature-complexity tasks.Increasing random-feature count helps only on the one-feature task, and never surpasses deep ME-IRL.Source: §D.2 · p. 24