PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking the Implications of Human Feedback for Preference Learning in Human-Robot Collaboration

Qiping Zhang, Kate Candon, Debasmita Ghose, Marynel Vázquez

arXiv:2609.13982v1cs.ROcs.AIcs.HC

TL;DR

Collaborative robots need to learn personal task preferences, but fixed rules for labeling unchosen actions can diverge from human judgments. IMPLIED uses fixed-rule implications as an initial guide and revises them as direct evidence accumulates; across evaluations, it predicted implications better than fixed-rule and LLM baselines and improved preference estimation and robot behavior.

  • Problem

    Fixed implication rules extend limited direct feedback to unchosen actions, but human judgments can differ from those rules, complicating preference learning.

    Source: §1 · pp. 1–2

  • Method

    IMPLIED uses fixed-rule implications as a prior, then increasingly relies on a Gaussian Process classifier trained on direct evidence to revise labels for unchosen actions.

    Source: §1 · p. 2

  • Results

    Across recorded interaction trajectories and a physical pizza-making study, IMPLIED predicted human implications better than fixed-rule and LLM baselines, approaching a human-label oracle, while reducing preference-estimation error and improving robot behavior.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work treats implication modeling as a learnable aspect of preference learning for high-level collaborative decision-making.

    Source: §1 · p. 2

    Experiments assume structured tasks, known goal rewards, and relatively stable preferences; open-ended collaboration and systematically misleading feedback remain challenges.

    Source: §7 · p. 9

Abstract

from arXiv · show

In Human-Robot Interaction, the standard approach to learn a reward model that represents human preferences for robot behavior consists of three steps. First, the robot collects limited direct evidence from human feedback (e.g., positive or negative binary feedback). Then, the robot utilizes the direct evidence to derive accepted or rejected labels to feasible but unchosen actions using fixed implication rules. Finally, the robot updates the reward model with both the direct and derived evidence. Unfortunately, the fixed rule can hinder preference learning: in a user study with two collaborative simulation environments, human-provided implication labels often differed from the standard fixed rule, and using the human labels substantially improved reward learning with the Preference Learning from Implicit and Explicit Feedback (PIE) algorithm. Consequently, we propose IMPLIED, an implication modeling method that treats fixed-rule implications as an initial guide while learning to infer and revise accepted and rejected action labels over time. Across evaluations on recorded human-robot interaction trajectories and a physical robot pizza-making study, IMPLIED predicts human implications more accurately than the fixed rule approach and LLM baselines, approaching the performance of a human-label oracle. In turn, IMPLIED reduces preference-estimation error and leads to robot actions that are more often rational with respect to a combined reward (which includes the true preference reward and a task-specific reward) compared to baselines. By learning to reason about the implications of human feedback, this work enables more faithful and efficient robot behavior adaptation during human-robot collaboration.

1 Introduction

Collaborative robots must learn both how to achieve task goals and how people prefer tasks to be done. This section argues that fixed implication rules can diverge from human judgments, and introduces IMPLIED, which revises implications as feedback accumulates.

1 Introduction

  • People have varied preferences about task details, so collaborative robots must learn both task goals and how users prefer tasks to be done.Preferences can concern action ordering, object placement, or division of labor.Source: §1 · p. 1
  • Human-provided implication labels often differed from fixed-rule labels, and using human labels improved preference learning in two simulation tasks.The study compared human judgments about unchosen actions with commonly used implication rules.Source: §1 · p. 2
  • IMPLIED starts with fixed-rule implications and uses accumulated direct evidence to revise accepted or rejected labels for unchosen actions.It increasingly relies on a Gaussian Process classifier trained on direct evidence and recomputes implications as feedback arrives.Source: §1 · p. 2, Figure 1 · p. 2
  • Across recorded interaction trajectories and a physical pizza-making study, IMPLIED predicted human implications better than fixed-rule and LLM baselines, approaching a human-label oracle.It also reduced preference-estimation error and yielded robot actions more often rational under a reward combining preferences and task goals.Source: §1 · p. 2
  • The paper frames implication modeling as a learnable part of preference learning and reports that fixed implications often diverge from human judgments.Its user study collected human labels for unchosen actions.Source: §1 · p. 2

2 Preliminaries: Learning Preferences Over a Human-Robot Collaboration

The setup models collaboration as joint human–robot decision-making under a shared reward that combines task goals with human preferences. PIE learns preference weights from direct feedback and derives labels for feasible but unchosen actions using fixed implication rules.

Collaborative Setup

  • The shared reward combines a task-specific goal reward with a human-preference reward, weighted by γ, and the robot estimates the preference weights.The interaction is represented as an MDP with simultaneous high-level human and robot actions.Source: §2 · p. 3

Preference Learning from Implicit and Explicit Feedback

  • PIE treats chosen actions as direct evidence, then derives accepted or rejected labels for other feasible actions as implications.The paper notes that prior preference-learning work does not treat this classification of unchosen actions as learnable.Source: §2 · p. 3
  • For explicit feedback, PIE labels the robot’s chosen action according to the button response and labels other feasible robot actions oppositely.A positive response accepts the chosen action and rejects alternatives; a negative response rejects the chosen action and accepts alternatives.Source: §2 · p. 3
  • For implicit feedback, PIE assumes the human acts optimally relative to the human reward, accepting the chosen human action and deriving labels for other feasible actions.The human’s action supplies direct evidence about preferences.Source: §2 · p. 3
  • PIE estimates preference weights by optimizing the log-likelihood of accepted behavior using a non-parametric belief distribution and gradient ascent.Its Boltzmann-rationality model treats the collaborator as the human for implicit feedback or the robot for explicit feedback.Source: §2 · p. 4

3 The Implications of Real Human Feedback in H-R Collaborations

Human-provided implications differed from fixed-rule labels, and the study found better preference learning with human implications, significantly so for sandwich-making outcomes. Participants supplied labels for unchosen actions across two collaborative simulation tasks.

Study procedure

  • Eighteen participants completed sandwich-making and block-sorting tasks, providing labels for unchosen feasible actions after evaluating the robot’s action.Participants completed three episodes per task, with different initializations.Source: §3 · p. 4

Implication inconsistency

  • Fixed-rule labels mismatched human labels on 0.27 ± 0.03 of sandwich-making timesteps and 0.41 ± 0.02 of block-sorting timesteps.Label inconsistencies were 0.21 ± 0.02 and 0.22 ± 0.01, respectively.Source: §3 · p. 4

Human implications improve preference learning

  • At the earliest task-completion step, sandwich-making L2 error was 0.65 ± 0.03 vs. 0.97 ± 0.10, and conflict proportion was 0.10 ± 0.01 vs. 0.25 ± 0.05 for Human Implication vs. Fixed-Rule.Both differences were statistically significant.Source: §3 · p. 5
  • Human Implication significantly reduced sandwich-making L2 error and conflict proportion; both measures were lower but not significantly so in block-sorting.Figure 2 summarizes preference-learning performance; its dashed line marks the earliest task-completion step per task.Source: Figure 2 · p. 5, §3 · p. 5

4 The IMPLIED Method for Inferring and Revising Implications

IMPLIED makes the labeling of unchosen actions learnable, combining fixed-rule implications with predictions from a classifier trained on direct evidence. It revises derived labels as new feedback arrives, with the mixture weighting guided by classifier uncertainty.

4 The IMPLIED Method for Inferring and Revising Implications

  • IMPLIED treats implications for unchosen actions as learnable labels, revising them whenever new feedback arrives.Each observation is assigned to the accepted or rejected set using its estimated probability of acceptance and a nominal threshold of 0.5.Source: §4 · pp. 5–6
  • IMPLIED records collaborator, feedback modality, timestep, and evidence type as metadata for reasoning about implications.Chosen actions are direct evidence; unchosen actions are derived evidence.Source: §4 · p. 5
  • For each unchosen action, IMPLIED mixes the fixed-rule implication with an acceptance probability from a Gaussian Process classifier trained on direct evidence.The fixed-rule component uses the direct evidence for the same state, feedback modality, collaborator, and timestep; it assigns the same label to the observed action or the opposite label to a different action.Source: §4 · pp. 5–6
  • An observation-specific weight balances the fixed-rule and learned probabilities, favoring the fixed prior when classifier uncertainty is high.The classifier's learned probability estimates whether an observation belongs in the accepted set.Source: §4 · p. 6

5 Experiments and Results

In simulation and physical collaboration, IMPLIED more accurately inferred human implications than non-human baselines and improved downstream preference-learning outcomes. In the physical study, it also required fewer explicit feedback instances and interaction steps than Fixed-Rule.

5.1 Evaluation on Human-in-the-Loop Simulation Data

  • The simulation evaluation reconstructed feedback sets from fixed states, actions, and direct evidence, comparing methods against participant-provided implication labels.Methods included Fixed-Rule, No-Prior, four LLM baselines with differing context, and a Human oracle; metrics covered implication F1, preference-weight error, and rollout conflict proportion.Source: §5.1 · pp. 6–7
  • In simulation, IMPLIED significantly lowered preference L2 error and conflict proportion versus non-human baselines in block-sorting, and did so for conflict proportion in sandwich-making.For sandwich-making L2 error, it was significantly lower than all non-human baselines except LLM-HB; Table 1 reports downstream performance at the earliest task-completion timestep.Source: Table 1 · p. 7, §5.1 · p. 8
  • At the earliest shared task-completion step in simulation, IMPLIED achieved the highest F1-score in both tasks and generally reached higher F1 faster than non-human baselines.The reported F1-scores were 0.84 ± 0.01 for sandwich-making and 0.87 ± 0.01 for block-sorting.Source: Figure 3 · p. 7, §5.1 · pp. 7–8

5.2 Evaluation in Real-World Human-Robot Collaboration

  • The physical evaluation used a within-subjects pizza-making study with 20 participants, counterbalanced Fixed-Rule and IMPLIED conditions, and online preference updates.Participants chose when to provide explicit feedback, while implicit feedback was received at every step.Source: §5.2 · p. 8
  • In the physical study, online learning with IMPLIED achieved significantly lower L2 error and conflict proportion than Fixed-Rule, while requiring fewer explicit feedback instances and fewer steps to completion.The physical-study table reports performance at the earliest shared task-completion step; explicit feedback instances were 34.10 ± 0.88 vs. 43.45 ± 1.05, and interaction steps were 40.05 ± 0.83 vs. 45.35 ± 1.15.Source: Table 2 · p. 8, §5.2 · p. 8
  • In the physical study, more participants said IMPLIED learned faster, required less teaching, made fewer mistakes, and was their preferred method for future use.The respective counts were 17/20, 16/20, 17/20, and 18/20.Source: §5.2 · p. 8

6 Related Work

Preference-learning research spans demonstrations, evaluative feedback, and methods that combine feedback types. Related work also derives labels beyond direct input, while this work learns and revises implications for feasible unchosen actions from interaction evidence.

6 Related Work

  • HRI preference learning has used demonstrations or evaluative feedback, and recent work combines demonstrations, queries, corrections, preferences, binary feedback, instructions, and implicit action feedback [40] [42].Other work maps feedback types to choices [16] or reasons about evaluative feedback alongside implicit feedback from human actions.Source: §6 · pp. 8–9
  • Related methods derive labels through trajectory-pair labeling, confidence-based pseudo-labeling, consensus-filtered relabeling, or synthetic preference rankings [43] [44] [45] [46].Loftin et al. model what absent feedback implies; this work instead revises labels for feasible unchosen actions from accumulated evidence.Source: §6 · p. 9

7 Limitations and Future Work

IMPLIED learns and revises implication labels from accumulated direct evidence, with results suggesting improvements over fixed-rule and large-model approaches. The work identifies boundaries in preference representations, feedback modalities, task settings, and action spaces; the accompanying simulation appendix describes the human and robot roles in its MuJoCo tasks.

7 Limitations and Future Work

  • IMPLIED learns and revises implication labels from accumulated direct evidence, and results suggest improvements in implication prediction, preference estimation, and robot behavior over fixed-rule or large-model approaches.The section describes the two MuJoCo simulation tasks as having fixed human and robot roles; the robot is autonomous while the human controls the other agent.Source: §7 · p. 9, A Details of the User Study in MuJoCo… · p. 14, Figure 5 · p. 14
  • The method assumes designers know relevant preference-feature dimensions and estimates their weights, limiting its use to tasks where designers can specify what humans may care about.Future work could jointly infer the preference representation and weights.Source: §7 · p. 9
  • The study focuses on binary explicit feedback and implicit feedback from human actions; demonstrations and language remain future modalities to explore.The passage notes unified learning formalisms exist for those modalities.Source: §7 · p. 9
  • The experiments used structured tasks with known goal rewards and relatively stable preferences; extending to open-ended settings may require richer action representations and robustness to noisy or inconsistent feedback.IMPLIED assumes a non-adversarial collaborator; systematically misleading feedback could produce an overconfident, incorrect GP, and detecting inconsistency or falling back to the prior remains open.Source: §7 · p. 9
  • The method assumes enumerable discrete high-level actions; continuous actions and VLA policies would require defining implications over trajectories.Collaboration trajectory boundaries are not obvious, unlike fixed presegmented trajectories; noncollaborative VLA settings may be a more tractable entry point.Source: §7 · p. 9

A.1 Study Protocol

The MuJoCo protocol collected explicit feedback and implication labels alongside participants’ own actions across sandwich-making and block-sorting tasks. Participants’ selected preferences remained consistent across their actions and evaluations, while the robot updated its preference estimate and acted using that estimate.

A.1 Study Protocol

  • Before each task, participants selected a true preference and used it consistently when choosing actions and evaluating the robot; the robot knew the goal reward but not the preference weights.The study compared fixed-rule implications with participant-provided implication labels while keeping the preference-learning method the same.Source: §A.1 · p. 14
  • At each timestep, participants chose an action, then gave binary feedback on the robot’s action and labeled implications for both robot and human unchosen actions.The robot selected an action according to its current policy, and later actions used the latest preference estimate.Source: §A.1 · p. 14, Figure 6 · p. 15
  • Participants completed both sandwich-making and block-sorting tasks, with three randomly initialized episodes per task and counterbalanced task order.Each episode continued until task completion; 18 participants completed the study, and sessions typically lasted 45 minutes.Source: §A.1 · pp. 14–15

A.2 Sandwich-Making Task

The sandwich-making task models collaboration over a target recipe using discrete actions, shared task state, and a reward combining goal completion with interpretable preference features. Participants selected one of six preferences and one of six recipe initializations for their trials.

A.2 Sandwich-Making Task

  • The task assembles a target sandwich from 16 ingredients across storage, workspace, sandwich, and the agents’ hands, with workspace capacity limited to four ingredients.The state and target recipe determine satisfied ingredient counts, item availability, whether either agent holds an ingredient, and whether workspace capacity is active.Source: §A.2 · pp. 15–16
  • Each agent has 19 discrete actions, including ingredient picks, puts to available locations, and WAIT; human and robot actions occur simultaneously.The robot moves ingredients between storage and workspace, while the human moves them between workspace and sandwich.Source: §A.2 · p. 16
  • Human preferences use a 10-dimensional weight vector: six dimensions encode ingredient-order preferences and four encode workspace-occupancy preferences.Positive ordering weights specify one ingredient type before another; positive workspace weights prefer workspace size at most the corresponding threshold.Source: §A.2 · p. 16
  • Participants selected one of six preference options before the task and used it consistently across sandwich-making trials.The options cover ingredient-order preferences and preferences for more than two versus at most two ingredients in the workspace.Source: §A.2 · p. 16
  • Each environment initialization specified a target recipe, with six recipe options provided for the study.The options varied the counts and types of bread, cheese, meat, tomato, and cucumber ingredients.Source: §A.2 · p. 16
  • The shared reward combines a goal reward with a preference reward, with γ controlling their relative importance.The goal reward includes task-completion bonuses and step costs, plus penalties for redundant, regressive, or irrelevant actions.Source: §A.2 · pp. 16–17

A.3 Block-Sorting Task

The block-sorting task has nine blocks, two target bins, and a shared robot-human reward combining task goals with block-bin preferences. Its appendix also specifies the preference-learning setup and participant demographics.

A.3 Block-Sorting Task

  • The task sorts nine blocks, formed from three colors and three shapes, into red and green bins across five regions shared unevenly by robot and human.The robot can reach left, red, and mid; the human can reach mid, green, and right, with mid supporting handoff.Source: §A.3 · p. 17
  • States track block allocation across five regions and both hand locations; each agent has 15 actions, and the shared middle region enables handoff.Actions are PICK, PLACE, or WAIT, restricted to regions each agent can reach.Source: §A.3 · p. 17
  • Preference weights represent which bin each block type should reach: positive favors red, negative favors green, and near-zero permits either.Participants chose one of six color-based, shape-based, or mixed preference options and kept it across block-sorting trials.Source: §A.3 · pp. 17–18
  • The shared reward combines goal and preference terms, while nine-dimensional block features encode whether actions favor red or green placement.Goal rewards include a +10 completion bonus and a −0.3 step cost; middle-region actions are signed according to which agent can next make progress.Source: §A.3 · p. 18

A.4 Additional Preference-Learning Parameters

  • The study used one shared set of preference-learning parameters across collaboration tasks and implication sources to focus comparisons on the source of implication labels.The setup follows Candon et al..Source: §A.4 · p. 19, Table 3 · p. 19

A.5 Demographic Information

  • The study recruited 18 participants, whose mean age was 24 years (SD = 7.2) and mean robotics familiarity rating was 4.78 (SD = 1.40).Ages ranged from 18 to 50.Source: §A.5 · p. 19

B.1 Derivation of λ in Eq. (4)

IMPLIED weights its fixed-rule prior against a learned GP according to the GP’s uncertainty. The appendix explains this weight and describes participant-held-out hyperparameter selection for simulation and real-world experiments.

B.1 Derivation of λ in Eq. (4)

  • The mixture weight λ(o) is small when the learned GP probability is near 0.5 and large near 0 or 1, reflecting uncertainty about the binary implication label.The Bernoulli variance is maximal at p = 0.5, with vmax = 0.25; normalizing by this maximum yields a weight bounded in [0, 1].Source: §B.1 · p. 19

B.2 GP Hyperparameter Search

  • The GP covariance matrix uses Z-score-normalized observation features, with a linear kernel scale and additive bias variance.The latent function values follow the GP prior induced by this kernel.Source: §B.2 · p. 20
  • For simulation, leave-one-participant-out validation selected GP hyperparameters using cumulative direct evidence from training participants and implication-prediction performance on validation participants.The held-out participant’s implication labels were never used for hyperparameter selection.Source: §B.2 · p. 20
  • For real-world collaboration, all participants used the parameter setting most frequently selected across simulation folds: σ2_b = 0.2.This setting was used for every participant in the real-world study.Source: §B.2 · p. 20

B.3 Sensitivity to the Threshold τ

The appendix tests the binary decision threshold and compares the GP with a simpler KNN learner. At the tested shared task-completion step, GP outperformed KNN across both tasks’ reported metrics.

B.3 Sensitivity to the Threshold τ

  • For threshold τ, 0.5 significantly outperformed both alternatives on all three sandwich-making metrics and outperformed both alternatives on block-sorting F1.In block-sorting, τ = 0.5 also outperformed τ = 0.25 on L2 error; no significant difference from τ = 0.5 was found for τ ∈[0.40, 0.60].Source: §B.3 · p. 20, Table 4 · p. 20

B.4 Comparison with a Simpler Learned Classifier

  • At the earliest shared task-completion step, GP outperformed KNN on F1, L2 error, and conflict proportion in both tasks.The task-specific metric values are reported below.Source: §B.4 · pp. 20–21
  • On sandwich-making, GP beat KNN on F1 (0.84±0.01 vs. 0.75±0.03), L2 error (0.71 ± 0.06 vs. 0.89 ± 0.07), and conflict proportion (0.13 ± 0.01 vs. 0.19 ± 0.03).The reported differences were significant for all three metrics.Source: §B.4 · pp. 20–21
  • On block-sorting, GP beat KNN on F1 (0.87 ± 0.01 vs. 0.71 ± 0.03), L2 error (0.42 ± 0.07 vs. 0.60 ± 0.08), and conflict proportion (0.12 ± 0.03 vs. 0.20 ± 0.05).The reported differences were significant for all three metrics.Source: §B.4 · p. 21

C Learning with Human-in-the-Loop Simulation Data

This section examines IMPLIED’s use of human feedback to label unchosen actions and support preference learning, covering simulation comparisons with Fixed-Rule across explicit- and implicit-only feedback as well as LLM-baseline prompts. It also outlines the physical pizza-making study setup and robot roles.

C Learning with Human-in-the-Loop Simulation Data

  • The simulation evaluation used leave-one-participant-out cross-validation, holding one participant out for testing and splitting the remainder into training and validation participants.The training and validation data were used to fit hyperparameters, including for the GP in IMPLIED and the No-Prior baseline.Source: C Learning with Human-in-the-Loop… · p. 21
  • With explicit-only feedback, IMPLIED reached the highest earliest-task F1 in both tasks: 0.86 ± 0.02 in sandwich-making and 0.84 ± 0.01 in block-sorting.For downstream learning, IMPLIED had significantly lower L2 error than all non-human baselines except No-Prior and lower conflict proportion than all such baselines in sandwich-making; in block-sorting, it significantly outperformed all non-human baselines on both metrics.Source: Figure 7 · p. 21, Table 5 · p. 21, C Learning with Human-in-the-Loop… · p. 22
  • With implicit-only feedback, block-sorting results favored IMPLIED: it achieved F1 0.86±0.01 and significantly lower downstream L2 error and conflict proportion than all non-human baselines.The F1 advantage was significant at the earliest task-completion step (p < 0.0001); downstream differences were significant at p < 0.05.Source: Figure 8 · p. 21, Table 6 · p. 21, C Learning with Human-in-the-Loop… · p. 22
  • In sandwich-making with implicit-only feedback, IMPLIED underperformed Fixed-Rule on F1; L2 error was 0.84 ± 0.05 versus 0.86 ± 0.06 (p = 0.3945).Human implicit-feedback actions were more constrained than in block-sorting, and labels had 0.04 ± 0.01 timestep and 0.03 ± 0.01 label inconsistencies against the fixed rule.Source: C Learning with Human-in-the-Loop… · p. 22

D Prompts for LLM Baselines

  • The LLM prompt condition LLM-HB included interaction history and a preference-belief summary; the other baselines removed one or both information sources.Prompts asked the model to label unchosen robot and human actions as accepted or rejected and return strict JSON.Source: D Prompts for LLM Baselines · p. 22
  • LLM-HB prompts provided task context, collaboration history, and current preference beliefs, then requested accepted-or-rejected labels for each unchosen action based on task progress and inferred preference.The prompts required unique action keys and strict JSON output.Source: D Prompts for LLM Baselines · pp. 29, 30, 35

E.1 Study Setup and Procedure

  • The pizza station used storage, workspace, and on-pizza zones, with a shared four-item capacity for the latter two that required clearing an item before another ingredient could be passed.The robot selected ingredients from storage and placed them in the workspace; the participant chose which ingredients to use and when.Source: §E.1 · p. 35, Figure 9 · p. 36
  • The system paired a Franka Emika Panda arm for ingredient manipulation with the Shutter tabletop robot for speech and gaze, coordinated by behavior trees.The arm’s pick-and-place motions were generated through MoveIt Task Constructor.Source: §E.1 · p. 36
  • Participants used green and red buttons for positive and negative feedback, based on task progress and whether the robot respected their pizza-making preference.They could provide feedback whenever useful rather than at every step.Source: §E.1 · p. 36
  • A practice task familiarized participants with pizza-making, handoffs, workspace capacity, and feedback buttons, and fitted individual rationality coefficients for subsequent tasks.The practice preference was to use cheese before sauce; the fitted coefficients were for inferring preferences from implicit and explicit feedback.Source: §E.1 · p. 36
  • The study counterbalanced the order of preference learning with IMPLIED and the Fixed-Rule implication across participants.Both conditions used preference learning per Eq. (1).Source: §E.1 · p. 36
  • Participants completed two sequential pizza-making tasks per condition, using the same recipes while resetting the robot’s learned preference estimate between conditions.They were not told which implication method was in use; after both conditions, they answered comparison questions about the methods.Source: §E.1 · p. 36

E.2 Pizza-Making Task Specification

The pizza-making task combines recipe completion with learning a participant’s ingredient-ordering and workspace-occupancy preferences. Participants also reported their demographics and robotics familiarity.

E.2 Pizza-Making Task Specification

  • IMPLIED’s pizza task used 16 ingredients across sauce, cheese, meat, and veggie classes, with recipes specifying target ingredients and quantities.The robot accessed storage and the workspace, while the human accessed the workspace and on-pizza zone; participants knew their preference, but the robot did not.Source: §E.2 · pp. 36–37
  • The robot had 19 actions and the human 33, and both acted simultaneously to move ingredients among storage, workspace, hands, and the pizza.Robot actions included picking, putting ingredients in storage or the workspace, and waiting; human actions included picking, placing on the pizza, and waiting.Source: §E.2 · p. 37
  • Human preferences were encoded in an eight-dimensional, unit-normalized weight vector covering six ingredient-ordering relations and two workspace-occupancy preferences.The preference reward is linear in robot and human feature vectors, whose values indicate whether actions support the selected ordering or occupancy preference.Source: §E.2 · pp. 37–38
  • Before the main trials, participants selected one of six true preference options and used it consistently while acting and evaluating the robot.The options included ingredient-ordering and workspace-occupancy preferences.Source: §E.2 · p. 37
  • The shared reward combines goal and preference terms, with γ controlling their relative importance.Goal rewards encourage useful staging and recipe completion, while penalizing actions that do not advance the recipe; human goal rewards encourage adding needed ingredients.Source: §E.2 · p. 38

E.3 Demographic Information

  • Participants were 20 people, with a mean age of 26 and mean robotics familiarity of 4.6 on a 7-point scale.The age range was 21–31 years (SD = 2.1); 16 participants were male, 3 female, and 1 non-binary/third gender.Source: §E.3 · p. 38
Loading 2609.13982v1…