PaperlayerView exact revision on arXiv

Source-linked AI summary

Behavioural Quality Is Not Learning Utility: Objective Mismatch in Human Behaviour Models for Robot Learning

Haoyang Jiang, Dana Kulic, Michael Burke, Pamela Carreno-Medrano

arXiv:2610.34839v1cs.RO

TL;DR

Human behaviour models are used to create simulated interactions for robot RL, but behavioural similarity may not indicate how useful they are for policy learning. The paper compares common evaluation metrics with downstream policy performance across highway driving and object handover. No metric consistently tracks performance across domains and trajectory representations, and training against one model need not yield the best policy when evaluated against that model.

  • Problem

    Human behaviour models are often evaluated by resemblance to human data, while their usefulness as training partners for downstream robot policy learning is less commonly assessed.

    Source: §I · p. 1

  • Method

    The study compares behavioural evaluations of human behaviour models with RL policy performance across highway driving and object handover.

    Source: §VII · p. 11

  • Results

    No evaluation metric consistently identified models leading to better downstream policy performance across the two domains and trajectory representations.

    Source: §VI · p. 10, §VII · p. 11

  • Takeaways & Limitations

    Behavioural evaluation and downstream training utility are distinct, so a model favoured by existing metrics is not necessarily most useful for robot learning.

    Source: §VII · p. 11

Abstract

from arXiv · show

Human behaviour models are increasingly used to provide simulated human interactions for training robots via reinforcement learning (RL). Existing approaches primarily evaluate these models according to how closely their generated behaviours resemble real human data, while their usefulness as training partners is less commonly assessed. This creates a potential objective mismatch: a model that performs well according to behavioural evaluation metrics may not necessarily provide useful training experiences for downstream robot policy learning. Across two human-robot interaction (HRI) domains, highway driving and object handover, we evaluate different types of human behaviour models using common generative-model evaluation metrics. We consider multiple human trajectory representations and compare the resulting evaluations with the downstream performance of RL policies trained using the corresponding models. We find that existing behavioural evaluation metrics do not consistently predict downstream policy performance across domains and trajectory representations, while their evaluations are sensitive to the choice of trajectory representation. We also show that even when the true testing behaviour distribution is known, training against a different distribution may yield better downstream performance than training directly against the true distribution.

I. INTRODUCTION

Human behaviour models are used as scalable interaction partners for robot RL, but their resemblance to human data may not reflect how well they support policy learning. This study asks whether existing model metrics predict downstream utility across HRI tasks and representations.

I. INTRODUCTION

  • The study compares behavioural metrics and RL policy performance for rule-based and data-driven models across highway driving and object handover.Source: §I · p. 2
  • The authors report a mismatch between behavioural similarity and downstream RL utility, with metric predictiveness varying across domains and trajectory representations.Source: §I · p. 2
  • Model evaluation is sensitive to trajectory representation, with metric scores and rankings varying substantially, particularly across learned feature configurations.Source: §I · p. 2

A. Reinforcement Learning (RL) in HRI

B. Human Behaviour Modelling for Robot Training

C. Objective Mismatch in RL

  • Objective mismatch arises when conventional model-evaluation objectives do not reflect downstream control performance; whether this applies to HRI behaviour models remains open.Source: C. Objective Mismatch in RL · pp. 2–3

D. Evaluation Metrics for Generative Models

Generative-model evaluation commonly measures statistical similarity to reference data, but those metrics do not directly measure usefulness for robot learning. The paper evaluates human behaviour models using fidelity and diversity metrics and compares these assessments with downstream RL policy performance.

D. Evaluation Metrics for Generative Models

  • Generative-model metrics such as Fréchet Distance, precision and recall, and density and coverage assess sample similarity, fidelity, diversity, or distributional coverage, often in latent space.Source: D. Evaluation Metrics for Generative… · p. 3
  • Existing human-behaviour metrics assess resemblance to real data but do not account for how generated samples influence robot task performance; no widely adopted utility metric exists.Source: D. Evaluation Metrics for Generative… · p. 3

III. PROBLEM FORMULATION AND OVERVIEW

  • The study compares generated and real behaviours using fidelity metrics FD and density and diversity metrics APD, coverage, and MMS, then relates them to RL policy performance.Source: §III · p. 4
  • Evaluation spans naturalistic highway driving and close-proximity object handover, using highD and handover@RPL datasets, respectively.Source: §III · p. 4

IV. EXPERIMENT SETUP

The experiments train and evaluate human behaviour models on separate data subsets in highway driving and object handover, then use the training models in RL environments. The two tasks differ in interaction setting and policy objective.

1) Highway Driving

  • Highway models use the first 40 highD recordings for training and the remaining 20 to fit corresponding evaluation models; Deep Q-Learning trains an ego vehicle among model-controlled traffic.Source: 1) Highway Driving · p. 4

2) Object Handover

  • Handover models use 1,281 preprocessed giver trajectories from 12 right-handed participant pairs, split into six training pairs and six evaluation pairs.Source: 2) Object Handover · p. 4
  • In the handover RL environment, the model controls the giver’s hand while the agent moves the taker’s hand toward it, earning an additional reward for successful completion.Source: 2) Object Handover · p. 4

B. Human Behaviour Models

The study compares rule-based and data-driven human behaviour models in highway driving and object handover, generating trajectories under consistent domain-specific simulation conditions. The collected behaviours include substantial differences in realism and driving safety.

B. Human Behaviour Models

1) Rule-based Driving Models:

  • Highway rule-based models combine MOBIL lane-change decisions with Gipps, Krauß, or IDM car-following models, whose parameters are fitted to highD data.Source: 1) Rule-based Driving Models · p. 5

2) Data-driven Driving Models:

  • Highway data-driven models include deterministic Behaviour Cloning, multimodal MDN, and an attention-based model incorporating temporal and interaction information.Source: 2) Data-driven Driving Models · p. 5

3) Rule-based Giver Models:

  • Handover rule-based giver models generate either smooth straight-line minimum-jerk movements or elliptical trajectories; their parameters are estimated from demonstrations and sampled.Source: 3) Rule-based Giver Models · p. 5

4) Data-driven Giver Models:

  • Handover data-driven models include probabilistic DMP and reactive MLP, MDN, and attention models that predict the giver’s next hand displacement from both hands’ recent positions.Source: 4) Data-driven Giver Models · p. 5

C. Trajectory Collection

D. Trajectory Representations

The study compares PCA, autoencoder-latent, and manually extracted trajectory representations, pairing distributional or feature-wise metrics with cross-evaluated RL policy performance. It also uses regret to compare metric-based model selection with random selection.

D. Trajectory Representations

  • Three trajectory representations are evaluated: PCA components, an autoencoder latent space, and manually extracted behavioural features.Manual features include speed and acceleration for both domains, plus domain-specific measures such as headway in driving and smoothness in handover.Source: D. Trajectory Representations · p. 6

E. Evaluation Metrics

  • Evaluation uses representation-based metrics for PCA and latent spaces and feature-wise metrics for manually extracted behavioural features.The representation-based metrics characterize distribution fidelity and diversity in lower-dimensional spaces whose features may lack direct physical interpretations.Source: E. Evaluation Metrics · p. 6, 1) Representation-Based Metrics · p. 6

1) Representation-Based Metrics:

  • PCA and latent representations are assessed with distributional metrics that characterize generated-feature fidelity and diversity relative to human data.Their dimensions may not have explicit semantic or physical interpretations.Source: 1) Representation-Based Metrics · p. 6

2) Feature-Wise Metrics:

  • For manually extracted features, KL divergence compares generated and human distributions; lower divergence indicates a closer match.The study computes KL divergence separately for each feature and averages across features as an aggregate measure.Source: 2) Feature-Wise Metrics · pp. 6–7

F. Experimental Procedure

  • Trained policies are cross-evaluated against every corresponding evaluation model, with average cross-evaluation reward used as overall downstream performance.This tests performance beyond the behaviour-model type used during training.Source: F. Experimental Procedure · p. 7
  • Each metric selects a training model, and its regret is compared with average regret from random selection; zero marks the best model and one the worst.The interpretation assumes model utilities mostly differ, an assumption the authors say holds in their experiments.Source: F. Experimental Procedure · p. 7

V. RESULTS

Some metrics select useful training models in particular domains and configurations, but none is robust across both domains and all tested settings. Evaluation scores also may not reflect whether generated behaviour is realistic or physically plausible.

V. RESULTS

  • MLP and MDN score well on fidelity and diversity metrics in highway driving but also have relatively high crash and drive-off rates.The evaluation metrics may capture some fidelity and diversity aspects without fully reflecting realistic or physically plausible behaviour.Source: §V · p. 7, Table IV · p. 8

A. Behavioural Evaluation Metrics Do Not Consistently Predict Downstream Utility

  • Density is a useful selector in handover with latent representations and highway with PCA, while several manual-feature KL metrics beat random selection in both domains.These successes are configuration- or feature-dependent rather than robust across all tested settings.Source: A. Behavioural Evaluation Metrics Do… · p. 8
  • No evaluated metric robustly predicts downstream utility across both domains, representations, and hyperparameter configurations.Table V summarizes normalized regret by domain, representation, and configuration.Source: A. Behavioural Evaluation Metrics Do… · pp. 8–9, Table V · p. 9

B. Training on the Test Distribution Does Not Necessarily Maximise Downstream Performance

Cross-evaluation shows that training against the test behaviour model does not necessarily maximize reward under that model. Across domains and trajectory representations, behavioural metrics do not consistently identify useful training models, and the conclusions can depend on how trajectories are represented.

B. Training on the Test Distribution Does Not Necessarily Maximise Downstream Performance

  • Against IDM, Gipps-, Krauß-, MLP-, MDN-, and Attention-trained policies each exceeded the IDM-trained policy’s 26.9 average reward.The cross-evaluation heatmaps also show that diagonal entries are not always the largest within their evaluation columns.Source: B. Training on the Test Distribution… · p. 9, Figure 2 · p. 10

VI. DISCUSSION

  • Across two domains and three trajectory representations, no evaluated metric consistently tracked downstream policy performance.Metric–utility associations changed substantially with learned latent representations; PCA and manually extracted features were comparatively more stable, but remained sensitive to configuration and domain.Source: §VI · p. 10
  • The asymmetric highway results suggest that challenging but structured behaviours may help, while excessively unrealistic behaviours may hinder learning; this pattern is less apparent in coordinated handover.The authors hypothesize that challenging behaviours may resemble adversarial training, but note that Unif and Unif-W performed poorly despite higher crash and drive-off rates.Source: §VI · pp. 10–11

VII. CONCLUSION

  • Behavioural evaluation and training utility are distinct, so a model rated favourably by existing metrics may not be most useful for robot learning.The authors recommend investigating evaluation methods that consider support for downstream learning rather than relying solely on reproduction of observed behaviour.Source: §VII · p. 11

APPENDIX A ADDITIONAL EXPERIMENTAL DETAILS

The appendix specifies the highway and handover training settings and details the rule-based driving models. It describes MOBIL lane changes and Gipps parameter inference, including how sparse data and nondifferentiable components shape the fitting procedure.

A. Highway Driving

  • The highway setup uses highD trajectories and trains Deep Q-Learning agents in highway-fast-v0, with surrounding vehicles controlled by the fitted behaviour models.The fast variant uses three lanes, 20 vehicles, and 30-second episodes.Source: A. Highway Driving · p. 13

B. Object Handover

  • The handover setup uses the giver’s reverse-handover interaction segment and 1,281 processed trajectories to train an agent that controls a point-mass taker hand.The environment bounds hand motion to spheres of radius 0.8 m and ends an episode when the hands are within 0.2 m or 60 steps elapse.Source: B. Object Handover · p. 13

A. Minimising Overall Braking Induced by Lane Changes (MOBIL) Model

  • MOBIL parameters use standard values rather than data fitting because completed lane changes are sparse and estimating them would compound modelling uncertainty.Only about 5,600 of roughly 14,000 identified lane changes are completed within the observed area.Source: A. Minimising Overall Braking Induced… · p. 14

B. Gipps’ Model

  • Gipps assumes desired-speed and collision-avoidance constraints, and fits safe margin and desired speed separately using a 25 m distance-headway split and ADVI.The appendix reports population-level ADVI hyperparameters in Tables VI and VII but the supplied passages do not provide their values.Source: B. Gipps’ Model · pp. 14–15

C. Krauß Model

The Krauß model is fitted through reaction time, while IDM fits desired velocity, safe time headway, and jam distance. The models differ in their dynamics and in which observations and parameters are used for inference.

C. Krauß Model

  • Krauß inference fits only reaction time, using observations with distance headway below 25 m and a hierarchical Bayesian formulation for driver variability.Other parameters are fixed to Gipps values, apart from the positive maximum-deceleration parameter.Source: C. Krauß Model · p. 15

D. Intelligent Driver Model (IDM)

  • IDM is a second-order model using velocity and acceleration, and fits desired velocity, safe time headway, and jam distance while fixing remaining parameters.Unlike Gipps and Krauß, IDM is fully differentiable, so all available observations are used for training.Source: D. Intelligent Driver Model (IDM) · pp. 15–16

E. Unif and Unif-W

The highway experiments include stochastic acceleration baselines and three data-driven behaviour models. MLP predicts actions directly, whereas MDN represents action uncertainty and multimodality with a regularised mixture distribution.

E. Unif and Unif-W

  • Unif and Unif-W sample longitudinal and lateral accelerations independently, with Unif-W allowing a wider lateral range than Unif.Both use the same longitudinal range; Unif uses U(−2, 2) laterally and Unif-W uses U(−6, 6).Source: E. Unif and Unif-W · p. 16

F. MLP

  • The MLP behaviour-cloning model maps ego-centric vehicle-state features to continuous actions using supervised learning on highD demonstrations.Its test loss is 0.62 after early stopping at epoch 88.Source: F. MLP · pp. 16–17

G. MDN

  • The MDN models conditional actions as a Gaussian mixture, with regularisation encouraging mixture-component usage and diversity.It uses 15 components and achieves a test loss of 1.88 after early stopping at epoch 48.Source: G. MDN · pp. 17–18

H. Attention

The attention model uses the current ego state to attend to its history and surrounding vehicles, then conditions a CVAE to generate control actions. This design incorporates temporal context and multimodal action predictions.

H. Attention

  • Cross-attention uses the current ego state to query ego history and surrounding-vehicle states, producing embeddings that form the scene representation.Surrounding states are represented as tokens across vehicles and time steps; the ego and interaction embeddings are concatenated before CVAE conditioning.Source: H. Attention · pp. 18–19
  • The CVAE conditions action generation on the scene embedding, using a latent sample from the prior at inference when the ground-truth action is unavailable.The training objective combines reconstruction loss with KL-divergence regularisation.Source: H. Attention · p. 18
  • The architecture is a simplified attention-based Trajectron++ variant, replacing recurrent modules with attention and omitting road maps and explicit agent categories.Vehicle type is implicitly encoded through vehicle length in the state representation.Source: H. Attention · p. 19
  • The attention-based model achieves a best test loss of 0.52, with early stopping at epoch 41.The reported architecture uses a history length of four and a CVAE latent dimension of 16.Source: H. Attention · p. 19

I. Model Validation

The calibrated models capture meaningful distributional characteristics of naturalistic driving and human handover, although variability and feature-level fit differ. The section also describes three rule-based handover models that generate trajectories with distinct movement assumptions.

I. Model Validation

  • Calibrated driving models generally capture central tendencies in highD features, with MLP, MDN, and Attention achieving lower KL divergence for several features.Variability still differs across features.Source: §I · p. 19, Figure 4 · p. 20, Table X · p. 20

A. Minimum Jerk Model

  • The Minimum Jerk model generates smooth, deterministic giver-hand movement between sampled start and target positions, with duration controlling its temporal evolution.Start positions, end positions, and lengths are sampled independently from Gaussian distributions fitted to human training data.Source: A. Minimum Jerk Model · pp. 19–20

B. Elliptical Model

  • PC3 variance is substantially smaller than PC1 and PC2, motivating a two-dimensional movement plane for fitting and generating elliptical trajectories.The fitted ellipse and sampled trajectory geometry are used to capture variation in endpoints, orientation, shape, and duration.Source: B. Elliptical Model · p. 21

C. Dynamic Movement Primitive (DMP)

  • The DMP combines a stable spring–damper system with a nonlinear forcing function to capture movement shape while adapting initial position, goal, and duration.Weights fitted across demonstrations are modelled with a multivariate Gaussian and sampled to generate stochastic trajectories.Source: C. Dynamic Movement Primitive (DMP) · pp. 21–22

D. MLP

The handover models use next-step prediction architectures, while validation finds that their fit to human feature distributions varies by feature. The supplied material also describes trajectory collection and PCA- and autoencoder-based representations used for evaluation.

D. MLP

  • The handover MLP uses five frames of giver and taker hand coordinates to predict the giver hand’s next-step displacement.The input is flattened for the MLP, which follows the highway model’s architecture and supervised training procedure.Source: D. MLP · p. 22

E. MDN

  • The handover MDN predicts a five-component Gaussian mixture over the giver hand’s next-step displacement, reducing components from the highway model’s 15.The reduction is intended to reflect lower behavioural complexity in handover.Source: E. MDN · p. 22

F. Attention

  • The handover attention model applies one multi-head self-attention layer to five-frame hand-coordinate sequences, then pools time and predicts the next displacement with an MLP.Its lightweight architecture is designed to capture dependencies across the observation window.Source: F. Attention · p. 22

G. Model Validation

  • DMP achieves the lowest KL divergence for path length, average speed, and average acceleration, while data-driven models lead on some individual features.Overall, calibrated models capture meaningful handover characteristics but retain distinct behavioural profiles.Source: G. Model Validation · p. 23, Figure 5 · p. 23, Table XII · p. 24

APPENDIX D TRAJECTORY GENERATION AND COLLECTION

A. PCA Representation

  • PCA projects human and generated trajectories into a shared lower-dimensional space, retaining six components for highway and four for handover in the reference configuration.The principal axes are fitted on collected human trajectories.Source: Appendix E · pp. 23–24, A. PCA Representation · p. 24

B. Latent Representation

  • A reconstruction-trained autoencoder maps state sequences to latent vectors, and its trained encoder extracts features for metric evaluation of human and generated trajectories.The highway encoder uses a 32-dimensional latent vector; the handover autoencoder is trained on handover data and adapted to six-dimensional frames.Source: B. Latent Representation · pp. 24–25

C. Manually Extracted Behavioural Features

The section defines feature-distribution comparisons and generative-model metrics, then specifies consistent within-domain RL training procedures for comparing behaviour models. The metric descriptions distinguish distributional similarity, coverage, diversity, and novelty.

C. Manually Extracted Behavioural Features

  • The manually extracted features are compared between human and generated data using KDE-estimated distributions and KL divergence.Driving features are speed, acceleration, headway, and relative preceding speed; handover features include speed, acceleration, smoothness, curvature, end-to-end length, and path length.Source: C. Manually Extracted Behavioural… · p. 25

A. Fréchet Distance (FD)

  • Fréchet Distance compares Gaussian feature distributions using their means and covariances; lower scores indicate generated data closer to real data.It is interpreted relatively when comparing generative models.Source: A. Fr´echet Distance (FD) · pp. 25–26

B. Density / C. Coverage

  • Density counts generated samples within local neighborhoods of real samples, while coverage measures the fraction of real samples near at least one generated sample.Higher scores indicate better alignment for density and broader support capture for coverage.Source: B. Density · p. 26, C. Coverage · p. 26

D. Average Pair Distance (APD)

  • Average Pair Distance estimates dataset diversity by averaging Euclidean distances over randomly selected sample pairs.The procedure repeats random pair selection across multiple repetitions.Source: D. Average Pair Distance (APD) · p. 26

E. Mean Maximum Similarity (MMS)

  • MMS complements diversity measures with novelty by comparing generated-to-real similarity against real-to-real similarity, excluding self-comparisons.The novelty measure is the gap between those two MMS values.Source: E. Mean Maximum Similarity (MMS) · p. 26

F. KL Divergence

  • KL divergence compares KDE-estimated probability densities for each manually extracted feature between human and generated data.The section also defines an average across the manually extracted features.Source: F. KL Divergence · p. 26

A. Highway Driving

  • Highway policies use DQN with surrounding-vehicle behaviour set by a human model, holding the RL algorithm, policy, and hyperparameters fixed across training configurations.Policies are trained for 40,000 steps and evaluated against all human behaviour models.Source: A. Highway Driving · pp. 26–27

B. Object Handover

  • Handover policies use SAC with giver behaviour set by a human model, using the same training configuration across giver models and evaluation against six testing giver models.Each policy is trained for 100,000 steps and evaluated for 20 episodes per testing model.Source: B. Object Handover · p. 27

Related papers

Loading 2610.34839v1…