PaperlayerView exact revision on arXiv

Source-linked AI summary

Fixed-Reference Pose Residuals for Measuring Cross-Dataset Cue Transfer in Human-Robot Interaction Anticipation

Bowen Yang, Xinliang Xiao, Wenjing Zhang, Li Yang, Wei Zhou

arXiv:2610.12245v1cs.RO

TL;DR

Robots need to anticipate contact, but which cues transfer across robots and sites is unclear. The paper measures pose as a residual over frozen geometry; its gain is asymmetric across HUI360 and SSUP-A.

  • Problem

    It is unclear which anticipation cues retain their meaning when a model moves to another robot and site.

    Source: §1 · p. 2

  • Method

    FRPR freezes a source-trained geometry predictor and adds a zero-initialised temporal pose correction to its logit, making each prediction an exact reference-plus-pose split.

    Source: §1 · p. 2

  • Results

    AP gain was +0.044 (95% interval [+0.024, +0.082]) from SSUP-A to HUI360, with no measurable gain in reverse; asymmetry persisted over stronger source-selected geometry.

    Source: §7 · p. 16

  • Takeaways & Limitations

    Pose gains were asymmetric across transfer directions, while whether a person faced the camera retained its discriminative direction across datasets.

    Source: §7 · p. 16

    Source-chosen thresholds let geometry-using neural models detect at most 17% of target interactions; new-site threshold calibration was outside the study.

    Source: §6.2 · p. 14

Abstract

from arXiv · show

Social and service robots in public spaces need to anticipate which nearby person is about to approach and touch them, so that a response can be prepared before contact. It is largely unknown which cues support this anticipation when a model trained with one robot is used on another robot at a different site. We study this question with a fixed-reference pose residual (FRPR) model: a geometry predictor built from the person's bounding box and mask is trained and frozen, and a temporal network then learns from body pose an additive correction to its logit, so that every prediction splits exactly into a geometry term and a pose term. Between two public egocentric datasets recorded by different robots, HUI360 and SSUP-A, with every choice made on source data, the pose correction raised average precision (AP) from 0.277 to 0.321 from SSUP-A to HUI360 and gave no measurable gain in the opposite direction; the same asymmetry held over a stronger, source-selected geometry reference. Freezing gave no AP advantage over joint training, and simple geometric baselines and tree ensembles remained competitive or better, so the construction serves measurement rather than prediction accuracy. A head-orientation residual added small gains in both directions. Post hoc, whether a person faces the camera kept its discriminative direction across datasets, whereas head pitch reversed. With thresholds chosen on source data, the neural models that use geometry detected at most 17% of target interactions. Code and processed data are available at https://github.com/WeiZhou96/FRPR-interaction-anticipation.

1 Introduction

The section frames cross-robot interaction anticipation as a cue-transfer problem and introduces a frozen geometry reference with additive pose corrections to measure what transfers.

1 Introduction

  • Interaction anticipation predicts whether physical contact follows a short observation of a tracked person; the study measures geometry, pose, and head-orientation transfer between robots and sites.Source: §1 · pp. 2–3
  • Cross-robot transfer matters because camera placement and site conditions vary, while a fused network obscures which cue contributes to anticipation.Source: §1 · p. 2
  • FRPR freezes a source-trained geometry predictor and adds a zero-initialised temporal pose correction to its logit, making each prediction an exact reference-plus-pose split.Source: §1 · p. 2
  • AP rose by +0.044 (95% interval [+0.024, +0.082]) from SSUP-A to HUI360, with no measurable gain in reverse; the asymmetry persisted over a stronger source-selected reference.Source: §1 · p. 2
  • Permuted or coordinate-free pose controls showed no such gain, while freezing offered no AP advantage over joint training and simple geometric baselines and tree ensembles remained competitive or better.Source: §1 · p. 2

2 Related work

Prior work studies anticipation, attention cues, and multimodal classifiers; this paper distinguishes itself by retaining geometry as a fixed reference to measure cue transfer.

2.1 Anticipating interaction with a robot

  • Prior interaction-anticipation work used pose, motion, gaze, intent, and egocentric tracks; this study adopts HUI360 and the bidirectional evaluation of Lorenzo-Louis et al. [25].Source: §2.1 · p. 3

2.2 Head orientation as an attention cue

  • Engagement systems use face and head orientation as attention evidence, while this study measures camera-relative head orientation rather than gaze direction.Source: §2.2 · p. 3

2.3 Combining cues on a fixed base

  • Prior work reports branch overfitting and modality suppression; unlike a biased branch discarded at test time, this study keeps geometry as a deployed cue.Source: §2.3 · p. 3

3 Fixed-reference pose residual model

The model predicts interaction from event-anchored person tracks by adding a pose correction to a frozen geometry reference. Its staged design also supports an optional head-orientation correction and paired measurement of cue contributions.

3.1 Task and overview

  • Interaction anticipation is binary classification of 32-frame person tracks, with labels marking later physical contact and windows ending about one second before the event.Negative windows end the same lead time before the moment a non-interacting person appears largest.Source: §3.1 · p. 4

3.2 Input representation

  • Geometry summarizes box and mask position, size, and change over time; pose represents box-relative keypoints, confidence, velocity, and validity across frames.The geometry summary contains apparent motion but no articulation, while confidence and visibility in pose still vary with distance and viewpoint.Source: §3.2 · p. 4

3.3 Geometry reference and pose residual

  • A geometry MLP supplies a complete reference predictor, and a causal temporal pose network adds a zero-initialized logit residual while the reference remains fixed.The residual begins at zero, so the second stage starts with the geometry reference's predictions.Source: §3.3 · p. 5, Figure 1 · p. 5

3.4 Two-stage training

  • Stage 2 minimizes cross-entropy over the pose branch alone, keeping the geometry reference deterministic and without gradients.The study compares this frozen procedure with a matched joint model that differs by updating geometry parameters.Source: §3.4 · p. 5

3.5 What the fixed reference provides

  • The nested reference and FRPR models make the pose contribution a paired metric difference, but freezing does not guarantee better scores on new data.The frozen reference also remains available as a fallback if pose estimation fails; the summed model's relative branch scale is fixed.Source: §3.5 · p. 6

3.6 Controlled variants

  • Controlled variants test joint training, warm starts, dropout, scratch training, and nonlinear fusion of geometry and pose features.The fusion variants use a hidden layer of width 16, with one pose network parameter-matched to FRPR within 0.4%.Source: §3.6 · p. 6

3.7 Extension: a head-orientation residual

  • FRPR-H adds a source-selected head-orientation residual as a third stage, training it while the geometry and pose terms stay fixed.The orientation summary is derived from dense face keypoints and represents head orientation relative to the camera, not gaze.Source: §3.7 · p. 6

4 Experimental setup

The experiments transfer models between HUI360 and SSUP-A using date-based source splits and target-only evaluation. Model selection, thresholds, and orientation-cue screening are set using source data, with explicit checks on sampling and statistical limitations.

4.1 Datasets and sample pool

  • HUI360 and SSUP-A differ in robots, sites, crowd density, and interaction prevalence, and evaluation uses official source training pools and target test pools.The target interaction prevalence is 3.0% in SSUP-A and 16.7% in HUI360; the HUI360 test pool holds out environments, while SSUP-A uses the second field session.Source: §4.1 · p. 7
  • The official loader discards some negative tracks with missing pre-anchor frames, and SSUP-A azimuth describes image direction rather than the robot-relative interaction zone.A sensitivity analysis adds 1,128 negatives while leaving all 564 positives unchanged.Source: §4.1 · p. 7

4.2 Compared methods

  • Comparators include random forest and HistGB ensembles on geometry, pose, or both, plus a source-selected last-frame box-area ranking rule.Published HUI360 baselines are not compared numerically because they use a different sampling and normalization protocol.Source: §4.2 · p. 7

4.3 Model selection and statistical analysis

  • Hyperparameters and thresholds are selected on source data, with cross-validation folds formed by recording date; AP is primary and paired differences use date-resampled bootstrap intervals.Because the target prevalences differ, AP is not compared across transfer directions; intervals are approximate with five and nine target dates.Source: §4.3 · pp. 7–8
  • FRPR-versus-best-joint and best-fusion comparisons were protocol-locked, but other differences are exploratory and both target sets had been evaluated earlier in the project.No target information entered training, selection, or thresholding.Source: §4.3 · p. 8

4.4 Screening and locked evaluation of the orientation extension

  • Candidate cues were screened on source data against fixed pass criteria, then FRPR-H settings and checkpoints were locked before target head summaries were extracted.The locked decision rule required FRPR-H to improve mean AP in both directions, with at least four of five seeds positive and a confidence interval above zero in at least one direction.Source: §4.4 · p. 8

5 Results

The pose correction’s value depends on transfer direction: it improves AP from SSUP-A to HUI360 but not in the reverse direction. Neural models’ rankings transfer better than their source-selected operating thresholds, while simple models remain strong competitors.

5.1 Main comparison

  • AP is 0.321 for FRPR versus 0.277 for its reference from SSUP-A to HUI360, while both mean APs are 0.129 in the reverse direction.Pose-only models are close to chance in HUI360 →SSUP-A (AP 0.048 to 0.055 at prevalence 0.030), and as good as or better than geometry-only models in SSUP-A →HUI360.Source: §5.1 · p. 8
  • The six neural models using both inputs lie within a few hundredths of AP of one another, and FRPR is not the best in either direction.Their spread is of the order of seed-to-seed standard deviation.Source: §5.1 · p. 8
  • Simple models remain competitive: last-frame box area reaches 0.178 AP in HUI360 →SSUP-A, while HistGB with both inputs reaches 0.427 in SSUP-A →HUI360.Adding the head-orientation summary raises HistGB to 0.525 AP in SSUP-A →HUI360; a geometry random forest also exceeds the reference network there, 0.369 versus 0.277.Source: §5.1 · p. 8
  • At source-selected thresholds, geometry-using neural models detect at most 4% of target interactions in HUI360 →SSUP-A and 17% in SSUP-A →HUI360.Scores shrink under domain change, so source-appropriate thresholds become too strict even though rankings transfer better than thresholds.Source: §5.1 · pp. 8–9

5.2 Value of the pose correction over its reference

  • +0.044 AP (95% interval [+0.024, +0.082]): FRPR exceeds its reference from SSUP-A to HUI360; the reverse-direction difference is −0.0003 [−0.016, +0.018].The positive-direction gain remains between +0.040 and +0.054 when any one target date is removed; the reverse-direction leave-one-date-out differences range from −0.005 to +0.003.Source: §5.2 · p. 9

5.3 Freezing against joint training and fusion

Freezing the geometry reference does not improve AP over joint training under the tested comparisons. The component analyses show direction-dependent transfer, while the exact reference–correction split enables paired measurements of pose’s contribution.

5.3 Freezing against joint training and fusion

  • All four primary comparisons favor the jointly trained additive or fusion model by −0.009, −0.016, −0.005, and −0.005 AP, with every interval containing zero.Under matched optimizer settings, freezing neither improves cross-domain ranking nor shows a cost.Source: §5.3 · p. 9
  • FRPR is lower than the box-area ranking by −0.049 AP in HUI360 →SSUP-A, while HistGB with both inputs is higher by −0.106 AP in SSUP-A →HUI360.The latter interval excludes zero on the rebuilt pool and after refitting on all source data; tree-ensemble ordering varies by pool and refitting in the other direction.Source: §5.3 · p. 9

5.4 What the components reveal

  • The reference logit changes by up to 139 squared logit units under warm-started joint training while its ranking changes by at most 0.005 AP; FRPR’s reference change is zero by construction.For jointly trained models, the split z = g + r therefore has no fixed reading.Source: §5.4 · p. 9
  • On transfer to SSUP-A, the HUI360-trained reference retains AUROC 0.841 while pose falls to 0.650; in the reverse direction, pose holds 0.765 while the reference drops to 0.733.On source validation dates, the respective AUROCs were 0.923 and 0.911 for HUI360, and 0.745 and 0.871 for SSUP-A.Source: §5.4 · p. 9
  • From SSUP-A to HUI360, pose alone reaches 0.436 AP against 0.321 for the summed score, but its target standard deviation is 1.1 logit units versus 4.8 for the reference.The summed score largely follows the reference, and this target-data reading is exploratory; no combination weight was tuned or tested.Source: §5.4 · pp. 9–10

5.5 Sensitivity to lead time and to the content of the pose input

The pose gain from SSUP-A to HUI360 is not limited to one observation lead time and depends on the correct person’s keypoint positions. It also persists over stronger source-selected geometry references, though the stronger-reference analysis was post hoc.

5.5 Sensitivity to lead time and to the content of the pose input

  • Across windows ending 1.07 to 3.20 seconds before onset, FRPR’s gain is positive at every lead time from SSUP-A to HUI360 and remains close to zero in the reverse direction.Intervals are above zero at three of five lead times in the positive-gain direction; the result uses 27 common-track interactions.Source: §5.5 · p. 10
  • Permuting pose among same-date samples removes the SSUP-A →HUI360 gain, and zeroing keypoints and velocities also removes it; omitting only explicit velocity channels leaves a +0.028 gain.The fixed control configuration has an intact-model gain of +0.048 [+0.027, +0.091]; confidence-only input does not differ from the reference.Source: §5.5 · p. 10

5.6 Pose residuals on a stronger geometry reference

  • The stronger-reference analysis selected among an MLP, HistGB, and random forest using source-date cross-validation, then evaluated locked models on target data.Its arms and comparisons were fixed after earlier target results had been inspected.Source: §5.6 · p. 11
  • +0.086 AP [+0.054, +0.138]: the pose residual improves the selected reference from SSUP-A to HUI360, while the reverse-direction gain is +0.011 [−0.008, +0.023].The same directional pattern appears over a fixed HistGB reference: +0.090 [+0.046, +0.163] versus −0.006 [−0.016, +0.005].Source: §5.6 · p. 11
  • The selected-reference family was not stable across outer folds, and its source-selected thresholds detected only 0.04 and 0.12 of target interactions on average.In SSUP-A →HUI360, the fixed HistGB reference with its residual scored higher than the selected one by −0.050 [−0.091, −0.014].Source: §5.6 · p. 12

5.7 Head orientation as a further residual

Adding head orientation yields small AP gains in both transfer directions, but FRPR-H is not the model that exploits the cue best. Post hoc analysis finds that camera-facing cues retain their discriminative direction across datasets, whereas head pitch reverses.

5.7 Head orientation as a further residual

  • In source screening, adding head orientation raises HistGB AP by +0.039 on HUI360 and +0.047 on SSUP-A; permutation and confidence controls change it by at most +0.015.The gain requires the orientation values of the correct person; the hand summary and combined block also pass the screen.Source: §5.7 · p. 12
  • +0.011 AP [+0.003, +0.020] in HUI360 →SSUP-A and +0.007 [+0.001, +0.018] in SSUP-A →HUI360: FRPR-H improves over FRPR in both directions.Tree ensembles gain more from orientation; in SSUP-A →HUI360, HistGB with head orientation is clearly above FRPR-H by −0.196 [−0.316, −0.103].Source: §5.7 · p. 12
  • FRPR-H makes the orientation contribution readable exactly, but it is not the model that exploits the cue best.Fusion-H has higher point estimates in both directions, and HistGB with head orientation clearly exceeds FRPR-H in SSUP-A →HUI360.Source: §5.7 · pp. 12–13
  • The orientation term alone reaches AUROC 0.729 in HUI360 →SSUP-A and 0.760 in SSUP-A →HUI360, close to the pose term’s 0.765 in the latter direction.Its mean absolute value is 1.01 and 0.19 logit units, compared with 4.93 and 0.89 for the pose term.Source: §5.7 · p. 13
  • Post hoc, camera-facing cues retain their discriminative direction across transfers, while head-pitch cues reverse; 12 of 22 orientation entries keep their direction in both transfers.Target AUROCs for camera-facing entries range from 0.68 to 0.86 when oriented by the source sign.Source: §5.7 · p. 13

5.8 Individual corrections

Individual tracks show how pose residuals can move scores toward or away from their labels, but the residual signs are more consistent across seeds than their magnitudes.

5.8 Individual corrections

  • The displayed residuals move three tracks toward their labels and one away, but neither positive case crosses its source-selected threshold.The cases were selected by residual-sign frequency, not by whether the score moved toward the label.Source: §5.8 · p. 13
  • An interacting HUI360 track receives a +0.982 correction, raising its score but leaving it below threshold under every seed.The examples use seed-11 checkpoints fixed before case inspection.Source: §5.8 · p. 13
  • Residual signs persist across all five seeds for the displayed tracks, while residual magnitudes vary widely.The attributions largely follow displacement from a static reference and are not evidence about particular body parts.Source: §5.8 · p. 13

6 Discussion

Discussion of the two-robot, two-site evaluation finds cue transfer asymmetric, warns that source-selected thresholds transfer poorly, and frames the frozen reference as a measurement tool rather than an accuracy advantage.

6.1 Which cues a social robot can carry to a new platform

  • Geometry ranking transferred from HUI360 to SSUP-A, while pose behaved in the opposite direction; neither cue can be assumed to transfer generally.Facing-camera cues kept their discriminative direction across datasets, whereas head pitch reversed; the proposed explanations for pitch and yaw differences remain untested.Source: §6.1 · p. 13, Figure 4 · p. 14

6.2 What anticipation errors mean for the robot

  • Source-selected thresholds detected at most 17% of target interactions for neural models using geometry, despite above-chance target rankings.Target-site calibration was outside the study, and the lead-time analysis was too small to establish how early a robot can reliably act.Source: §6.2 · p. 14

6.3 The fixed reference as a measurement instrument

  • Freezing did not improve target rankings over joint training; it made pose's contribution measurable against a deployable, unchanged geometry reference.The construction can add cues without retraining earlier stages, but the results give no reason to prefer FRPR when only final ranking matters.Source: §6.3 · pp. 14–15

6.4 Limitations

  • The benchmark predicts physical contact rather than intention and does not estimate continuous-operation false alarms; its negative-window design may favor apparent size.Evaluations were not blind, and the five and nine target recording dates bound interval precision.Source: §6.4 · p. 16

7 Conclusion

The conclusion presents FRPR as a way to measure cue contributions across robots and sites, while finding asymmetric pose transfer and poor threshold transfer. Supplementary materials contain implementation, protocol, and detailed analyses.

7 Conclusion

  • Pose raised AP by +0.044 from SSUP-A to HUI360 and gave no measurable gain in the reverse direction, with the asymmetry also present over a stronger reference.Geometry and pose retained ranking quality in opposite transfer directions.Source: §7 · p. 16
  • Head-orientation residuals added small gains in both directions, and whether a person faces the camera kept its discriminative direction across datasets.Freezing did not improve AP over joint training, simple geometric baselines remained competitive, and source-selected thresholds transferred poorly.Source: §7 · p. 16

Supplementary Information.

  • The supplementary resource provides implementation and protocol details, per-seed results, sensitivity analyses, and the stronger-reference and cue-screen analyses.It also includes tree-ensemble, lead-time, sample-pool, case, and attribution analyses.Source: Supplementary Information · p. 16

Statements and Declarations

The declarations report funding and no competing interests, describe the use of publicly released datasets, and provide data, code, and AI-tool disclosures.

Funding.

  • The work was supported by four Jiangsu provincial and national science and technology grants.The listed awards include the Jiangsu Provincial Major Science and Technology Program and the National Natural Science Foundation of China.Source: Funding · p. 16

Competing interests.

Ethics approval and consent to participate.

  • The study used publicly released HUI360 and SSUP-A data and collected no new data involving human participants.SSUP-A is the annotation of the SSUP-HRI street recordings.Source: Ethics approval and consent to… · p. 16

Consent for publication.

  • The publication statement is not applicable.The cited figure materials come from publicly released HUI360 data, and no new images of individuals were collected.Source: Consent for publication · p. 16

Data availability.

  • Processed data, per-sample predictions, and derived statistics are publicly available on GitHub.Source: Data availability · p. 16

Code availability.

Use of AI tools.

  • Claude was used to polish the manuscript's English, and the authors reviewed and edited the text.The authors state that they take responsibility for the publication's content.Source: Use of AI tools · p. 17

ORCID.

  • ORCID identifiers are listed for Bowen Yang, Xinliang Xiao, Wenjing Zhang, Li Yang, and Wei Zhou.Source: ORCID · p. 17
Loading 2610.12245v1…