Source-linked AI summary
Embodiment-aware control by inference over the operator: a simulation study
Sara Falcone
TL;DR
Teleoperation systems optimize channel fidelity, while Sense of Embodiment is usually assessed only after use and the operator’s predictions are unobservable. The paper’s UEE infers hidden embodiment and cue weighting from implicit signals and task outcome to choose device settings under explicit preferences. In simulation it approached oracle-level embodiment, while comparisons identify inference over the operator and explicit preferences—not information seeking alone—as central to the benefit.
Problem
Teleoperation optimizes channel fidelity, but Sense of Embodiment is assessed only afterwards and the operator’s predictions needed for prediction-error control are unobservable.
Source: §1 · pp. 1–2Method
Results
1.68 for the UEE against 0.73 for fixed control; most operators had reached a suitable setting before their exact type was identified.
Source: §7 · pp. 10–11Takeaways & Limitations
In simulation, Bayesian inference over the operator with explicit preferences comes close to oracle performance when the model is right, while failures point to calibrated signals and a model of operator change as needed measurements.
Source: §7 · p. 11The main comparison is partly built into the design: synthetic operators come from the controller’s model family, and fixed control suits only balanced operators.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Teleoperation systems are tuned for channel fidelity, while whether the operator experiences the device as part of the body, the Sense of Embodiment (SoE), is measured only afterwards, by questionnaire. Predictive-processing accounts suggest controlling devices to reduce the mismatch between the operator's predictions and the returned feedback, but those predictions are unobservable, and an objective that only penalizes mismatch is minimized by removing feedback. We formulate an embodiment-aware controller, the Universal Embodiment Engine (UEE), that infers the operator's embodiment and visuo-proprioceptive cue weighting from implicit gaze and pupil signals and task outcome, and chooses bounded device settings under explicit preferences, cast as a discrete Active Inference agent. In simulations with 300 heterogeneous synthetic operators, the UEE found the suitable setting within half a minute for most operators, before identifying their exact type, and came close to an oracle in the second half of the session (embodiment 1.68 against 0.73 for the best fixed setting, on a 0-2 scale). Adapting without reading the operator did no better than fixed control, and model-free bandits did worse, whereas an expected-utility controller with the same inference did exactly as well: the benefit comes from Bayesian inference over the operator with explicit preferences, not from the information-seeking term of Active Inference. A naive prediction-error minimizer withheld feedback, as its objective implies, and lost task success (0.74 vs 0.90). The benefit shrank but persisted for operators outside the controller's model family, grew with the variety of the population, and vanished when the controller trusted an uninformative signal or when cue weighting changed mid-session without being modeled. These failures show what studies with people must establish first: calibrated signals and a model of change.
1 Introduction
Sense of Embodiment is relevant to device learning and continued use, but device engineering typically optimizes channel fidelity and measures embodiment only after use. The paper proposes inferring the operator’s hidden embodiment and cue weighting from implicit signals and task outcomes, then controlling device settings under explicit preferences.
1 Introduction
- Teleoperation and assistive devices are typically tuned for mechanical transparency, while Sense of Embodiment is assessed afterwards by questionnaire.Embodiment is implicated in device learning and continued use; questionnaire reports are sensitive to expectation and suggestion.Source: §1 · p. 1
- Predictive Transparency aims to match returned feedback to the operator’s predictions, but those predictions are unobservable and mismatch-only control can minimize error by removing feedback.Embodiment can tolerate moderate inconsistencies between the device and the operator’s body.Source: §1 · p. 2
- The UEE infers hidden embodiment and cue weighting from implicit signals and task outcome, then selects bounded device settings under explicit preferences.Its signals include pupil diameter, gaze transition entropy, gaze–hand lead time, and look-ahead-time ratio; a feedback-gain floor and minimum dwell constrain actions.Source: §1 · p. 2
- The simulations test whether inference can identify cue weighting quickly, improve embodiment and performance against control baselines, and avoid prediction-error collapse.They also examine model mismatch; the simulations test controller logic, not whether the proposed signals are valid in people.Source: §1 · p. 2
- The paper contributes a Bayesian-inference formulation, a demonstration of naive prediction-error collapse, and simulation analyses of operator inference, robustness, and failure modes.The comparisons test whether gains come from inference over the operator rather than adaptation alone or information seeking.Source: §1 · p. 2
2 Related work
Prior work includes online device tuning, inference of user goals, and adaptive embodiment in virtual reality. The UEE combines these strands by using implicit signals to infer embodiment online and choose device settings.
Adapting assistive devices to the individual
- Human-in-the-loop optimization tunes devices to measured objectives, such as metabolic cost, while the UEE models hidden embodiment because it cannot be measured directly.The paper’s bandit baselines represent model-free adaptation of this kind.Source: §2 · pp. 2–3
Problem structure
- The problem structure contrasts type-specific best settings with Mechanical Transparency, the best single setting for the modeled population.The two extreme cue-weighting types need opposite mappings; mild types benefit from the extreme mapping on their side, but less.Source: Figure 1 · p. 3
Inferring the user’s hidden state
- The UEE applies shared-autonomy-style hidden-state inference to whether an operator experiences a device as part of the body and which senses they rely on.Shared-autonomy methods instead infer a user’s latent goal from their inputs.Source: §2 · p. 3
Adaptive embodiment
- Prior VR work adapted avatar properties to individual tolerance, while predictive-processing accounts had been brought to teleoperation but not, as far as the authors know, used to control the interface.The UEE combines these strands through online, model-based inference of embodiment from implicit signals to select device settings.Source: §2 · p. 3
3 The UEE as Active Inference over the operator
The UEE models embodiment, cue weighting, and device setting as states, observes implicit signals and task outcome, and updates beliefs to choose settings. Its model links settings to embodiment dynamics and cue weighting to the information carried by observations.
Hidden states
- The controller represents embodiment, visuo-proprioceptive cue weighting, and device setting as three state factors, with nine combinations of feedback gain and mapping.Cue weighting spans five types from vision-dominant to proprioception-dominant; the mapping has visual, mixed, and proprioceptive options.Source: §3 · p. 3
Dynamics
- A setting immediately changes the device state, while embodiment rises or falls probabilistically according to the mapping–operator match and feedback gain.Cue weighting is stable within a session in the base model; a lapse term allows embodiment to drop even under a suitable setting.Source: §3 · p. 4
Observations
- Five observed modalities include pupil diameter, gaze transition entropy, gaze–hand lead time, look-ahead-time ratio, and task outcome.Gaze–hand lead time directly carries cue-weighting information; task success depends on embodiment as well as gain, and neighboring types are hard to distinguish from gaze–hand lead time alone.Source: §3 · p. 4
Inference and action
- The controller updates beliefs each second and evaluates each admissible setting using expected free energy, which combines negative expected utility and expected information gain.Preferences favor task success and high-embodiment signatures; a gain floor and five-second dwell constrain settings and change timing.Source: §3 · p. 4
4 Simulation design
The study simulates heterogeneous operators, compares the UEE with fixed, yoked, inference, and model-free controls, and measures embodiment and task outcomes. The authors qualify the main comparison because the synthetic operators share the controller’s model family and fixed control suits only balanced operators.
Synthetic operators
- Synthetic operators share the controller’s model structure but vary in cue-weighting type, embodiment dynamics, lapse rate, and signal noise.Cue-weighting types are drawn uniformly from five possibilities, and each operator begins at the lowest embodiment level.Source: §4 · p. 5
Conditions
- The nine conditions include fixed Mechanical Transparency, yoked setting replays, UEE variants, a prediction-error minimizer, expected-utility control, model-free bandits, and an oracle.Yoked replays match the UEE’s amount and timing of setting changes without reading the tested operator; the oracle uses that operator’s true type.Source: §4 · p. 5
Outcomes and analysis
- Three hundred operators completed 120-step sessions; the primary outcome was mean embodiment over the last 60 steps.Secondary outcomes included task success, setting changes, time at the lowest gain, time to a suitable mapping, and cue-weighting inference accuracy.Source: §4 · p. 5
What the simulation can and cannot show
- The main comparison is partly built into the design because synthetic operators come from the controller’s model family and fixed control suits only balanced operators.The study tests this boundary using out-of-family operators and population-variety sweeps.Source: §4 · p. 5
5 Results
The UEE rapidly finds settings suited to operators and raises embodiment and task success over fixed or yoked control, while nearly matching an oracle. Its advantage comes from inference over the operator with preferences, while a naive prediction-error objective and some ablations expose important trade-offs.
5 Results
- Within 30 s, the suitable mapping was reached and kept for 89% of operators, before exact cue-weighting identification was complete.At 30 s, true-type posterior probability was 0.66; 57 of 79 type errors confused types calling for the same setting.Source: §5 · p. 5
- 1.68 vs 0.73: UEE’s last-60-s mean embodiment exceeded Mechanical Transparency, with task success 0.90 vs 0.82.The UEE–Mechanical Transparency embodiment difference was 0.95 (95% CI [0.87, 1.04]); Figure 2 also shows the UEE close to the Oracle across operator types.Source: Figure 2 · p. 6, §5 · p. 6, Table 1 · p. 7
- The UEE’s last-60-s embodiment was within 0.01 of the Oracle, while its whole-session gap was 0.10.The Oracle gap was −0.009 in the last 60 s and −0.10 over the whole session; 80% of the identification cost was paid in the first 30 s.Source: §5 · p. 6
- The expected-utility controller matched the UEE, while UCB reached 1.30 and Thompson sampling 0.85 in embodiment, versus the UEE’s 1.68.Task success was 0.80 for UCB and 0.75 for Thompson sampling, against 0.90 for the UEE; the expected-utility controller was identical to the UEE.Source: §5 · pp. 6–7
- The naive prediction-error minimizer achieved task success of 0.74 versus the UEE’s 0.90, as it favored lower conflict by withholding feedback.Information seeking without preferences did not find the suitable setting; either preference alone was indistinguishable from the full UEE, and the floor was unnecessary with UEE preferences.Source: §5 · p. 7, Table 2 · p. 8
- Removing the dwell raised setting changes from 1.5 to 3.4 per session, while last-60-s embodiment was unchanged.The dwell’s intended purpose could not be evaluated because synthetic operators did not adapt to the controller.Source: §5 · p. 7
6 Robustness and operators outside the model family
Robustness tests show that the UEE’s benefit depends on calibrated signals and modeled within-session change, and degrades with model mismatch or populations already served by fixed control.
Matched model and signal noise
- +0.92 [0.82, 1.03] was the UEE’s benefit over Mechanical Transparency with the matched model; noisier signals reduced it to +0.85 at noise 0.45.Noise slowed identification, while tested changes in pupil behavior, adaptation speed, and lapse rates barely affected controller choices.Source: §6 · p. 8, Table 3 · p. 9
Signal calibration
- +0.06 [0.01, 0.10] was the benefit when GHLT was uninformative but assumed informative; calibrating the observation model restored it to +0.64 [0.53, 0.76].A starting prior of 0.6 or 0.8 on the true type recovered most of the remaining benefit, to +0.85 and +0.89.Source: §6 · p. 8, Table 3 · p. 9
Within-session change
- +0.06 [−0.00, +0.12] was the benefit after cue weighting changed at 60 s when the controller assumed it was stable; a small drift term restored the full benefit to +0.91 to +0.96.With drift, embodiment returned to 90% of its pre-change level about 35 s after the change; expecting change incurred a small cost when none occurred.Source: §6 · pp. 8–9, Figure 3 · p. 9, Table 3 · p. 9
Operators outside the model family
- +0.52 [0.41, 0.63] was the UEE’s gain for continuous cue weighting, versus +0.96 [0.86, 1.06] in-family; it captured 84% rather than 99% of the Oracle’s gain.Combining continuous weighting with a triangular match function gave the smallest gain, +0.38 [0.28, 0.48], or 82% of the Oracle’s.Source: §6 · pp. 9–10, Figure 4 · p. 10
Population variety
- +0.84 [0.72, 0.96] was the gain with no balanced operators, falling to +0.26 [0.18, 0.34] at 80% balanced and −0.02 [−0.04, −0.01] when all were balanced.The UEE stayed within 0.03 of the Oracle across population mixes; the caption shows the fixed-control setting was recomputed for each population.Source: §6 · p. 10, Figure 4 · p. 10
7 Discussion
The discussion argues that embodiment-aware control depends on inferring the operator with explicit preferences, not merely adapting settings or seeking information. Its simulations also expose signal calibration, workload confounds, and within-session change as key boundaries for human studies.
7 Discussion
- 1.68 embodiment for the UEE vs 0.73 for fixed control; information-seeking without preferences reached 0.93.Source: §7 · p. 10
- The expected-utility controller matched the UEE, while model-free bandits fell well short; inference and explicit preferences, not information gain, carried the benefit.Source: §7 · p. 11
- After 30 s, the most probable type was wrong for a quarter of operators, yet most already had suitable settings; benefit ranged from +0.84 to none as population variety fell.Source: §7 · p. 11
- Human studies must calibrate trusted signals, distinguish embodiment from workload effects, and model within-session cue-weighting change.Source: §7 · p. 11
- The simulation tests controller logic rather than real signal–embodiment relations; its operators, states, signals, and settings remain simplified or hand-tuned.Source: §7 · p. 11