Source-linked AI summary
DOBI: Dynamic Opportunistic Body Input via Spare Joint Recruitment for Hands-Free XR
Rachel Kim, Xun Qian, Sang Ho Yoon
TL;DR
XR interaction remains difficult during hand-busy tasks because common techniques assume available hands or fixed input channels. This paper studies opportunistic spare-body movements and implements DOBI, which uses gaze, a trigger gesture, and runtime motion mapping for continuous 1D control. Across controlled and realistic dual-task evaluations, DOBI achieved baseline throughputs up to 2.62 bits/s and SUS = 84.2, while the authors identify limits in evaluation scale and higher-dimensional control.
Problem
XR systems often assume available hands, while existing hands-free approaches are tied to fixed channels or regions with contextual limitations.
Method
The paper conducts elicitation across six hand-busy scenarios and implements DOBI with gaze targeting, trigger-conditioned region recruitment, kinematic-subtree extraction, and runtime PCA mapping.
Results
Up to 2.62 bits/s throughput and SUS = 84.2 were reported across four body regions and realistic hand-busy activities, respectively.
Takeaways & Limitations
Spare-body motion is a promising source of on-demand XR input that adapts to which body regions remain available during hand-busy interaction.
Takeaways & Limitations
The dual-task study included only 6 participants across 3 scenarios, and the 1D formulation does not establish support for higher-dimensional control.
Abstract
from arXiv · showhide
Extended Reality (XR) systems are often most useful when users are engaged in ongoing physical tasks, yet current interaction techniques still largely assume the hands are available. We present opportunistic body input, an interaction paradigm that redirects continuous XR control to whichever available body region remains free in the moment. To investigate how users naturally coordinate these spare-body movements, we conducted an elicitation study across six hand-busy scenarios. We found that while users' preferred spare body regions shift dynamically based on physical constraints, the resulting spontaneous movements share a consistent, low-dimensional kinematic structure organized around a dominant principal axis. Building on these findings, we present DOBI (Dynamic Opportunistic Body Input), a real-time XR technique that uses gaze to target a UI element, a brief trigger gesture to identify the recruited spare body region, and the region's subsequent motion to drive continuous 1D control. A 1D Fitts' law study establishes the baseline motor performance of this paradigm across four distinct body regions, achieving throughputs up to 2.62 bits/s with an overall 5.0% error rate, and a dual-task usability study shows that DOBI supports reliable, low-effort control (SUS = 84.2) while users remain engaged in realistic hand-busy activities.
1 Introduction
XR interactions often assume users’ hands are available, limiting control during ongoing physical tasks. DOBI addresses this gap by recruiting whichever spare body region is free and mapping its motion to continuous 1D control.
- Current XR techniques rely heavily on hand gestures, which become awkward and fatiguing when users’ hands are occupied.
- Existing hands-free modalities provide fixed channels with contextual limitations, including noisy or socially inappropriate voice input and gaze fatigue during sustained control.
- Prior body-input systems demonstrate viable wrist, forearm, torso, and foot interaction but generally anchor controls to one predetermined anatomical site.
- Opportunistic body input instead maps interaction to spontaneous movements of whichever spare body region remains available.
- Users’ preferred regions vary with physical context, while movements within a selected articulation often share a low-dimensional kinematic structure.
- DOBI combines gaze targeting, a deliberate trigger gesture, and subsequent spare-body motion to drive continuous UI control, evaluated through Fitts’ law and dual-task studies.
2 Related Work
Prior hands-free and body-based systems establish that meaningful input can occur without the hands, but typically depend on fixed channels or body regions. This paper extends opportunistic interaction to dynamic recruitment within the body itself.
- Hands-free research has explored voice, gaze, facial or oral gestures, and EMG, each offering distinct activation and continuous-control capabilities.
- These modalities remain contextually constrained because each is tied to a single channel, such as noisy voice or fatiguing sustained gaze control.
- Body-based systems use wrist, forearm, torso, and lower-limb movements, confirming the body as a rich input space beyond the hands.
- Unlike systems optimized for one predetermined region, the paper asks whether one system can support whichever body region is currently available.
- The opportunistic framing adapts interaction to internal bodily availability, recruiting and releasing spare regions as task demands change.
3 Design Elicitation Study
The elicitation study examined spare-body movements during six hand-busy scenarios. Results show context-dependent region choice but convergent, predominantly 1D movement structure that supports runtime control mapping.
- The study investigated whether users naturally repurpose spare limbs during hand-busy tasks and whether their movements share a general kinematic structure.
- Twelve experienced XR users were tracked with a GalaxyXR headset and four VIVE Ultimate Trackers while proposing spare-body movements.
- Six scenarios varied posture, mobility, and hand constraints: cooking, cleaning, repairing, washing hands, walking, and sitting.
- Region choice: Preferred input region changed by scenario: the right wrist, rated M = 4.78 overall, ranked first in only two of six scenarios.
- Articulation families: Participant articulations clustered into 11 recurring families, suggesting that selected body regions support a small set of anatomically favored continuous motions.
- Pose linearization: PCA captured a median 97.6% of variance in one component, with median straightness S = 0.95 and median linearity R^2 = 0.98.
- Calibration: Movement range varied by region, with wrist median PC1 span 1.09 versus head 0.45, while polarity agreement reached 92%–97% across four regions.
- Design goals: These findings motivate user-initiated recruitment, PCA-based axis extraction, deliberate activation, dynamic gain calibration, implicit polarity, and body-centric feedback.
4 The DOBI System
DOBI turns a user-selected spare-body movement into continuous XR control through gaze targeting, deliberate triggering, motion execution, and dwell-based release.
- DOBI is a real-time XR technique that converts spare-body movement into continuous control signals.
- Trigger: The user looks at a target UI element and performs a deliberate double-repetition trigger with the chosen spare-body movement.
- Control execution or discard: After detection, DOBI displays a mirrored skeleton highlighting the recruited limb and its motion range for continuous UI manipulation or accidental-interaction discard.
- Commit and release: The user stops at the desired value, then dwells at the target posture for 0.3s to commit and automatically disengage.
- Interaction lifecycle: The same 0.3s dwell threshold is used for both discarding and committing interactions.
4.1 Gaze and Trigger
DOBI uses a brief double-repetition movement to identify an available body region while the hands remain occupied, then detects the trigger and active root joint in real time. The ST-GCN achieved high trigger-detection and root-identification accuracy on held-out data.
- Trigger Design: A quick double-repetition movement both signals the transition to DOBI and hints at the intended control motion.The design adapts the familiar double-click temporal signature to spare-body movements such as wrist flicks or shoulder shrugs.
- Trigger Dataset: A dataset of 3916 samples captured deliberate triggers alongside hand-busy-task motions, idle movements, and single-repetition pulses.The recordings came from 12 participants across behavioral and social hand-busy scenarios.
- Temporal Detection: The 1.5 s rolling detection window covers the observed trigger-duration distribution, whose mean was 1.06 s and 95th percentile was 1.52 s.Wrist triggers were fastest, shoulders and head followed, and elbows were slowest.
- ST-GCN Pipeline: The ST-GCN continuously monitors upper-body skeletal data with separate heads for binary trigger detection and root-joint identification.The pipeline identifies the active root joint before extracting the corresponding motion structure for control.
- Model Evaluation: 98.5% binary detection accuracy, 98.6% recall, and a 1.6% false-positive rate were achieved on held-out data.Root-joint identification reached 95.9% average accuracy across eight classes, with rare errors between anatomically adjacent joints.
- Model Evaluation: The core ST-GCN uses 0.51M parameters, supporting deployment on standalone XR headset processors.The architecture is smaller than the original ST-GCN and other cited XR deep-learning models.
4.2 Motion-to-Control Mapping
DOBI converts the recruited body region’s trigger motion into a one-dimensional control axis and dynamically calibrates its usable range. PCA-based projection and quality gates determine whether the motion is sufficiently linear and broad for continuous UI control.
- Motion Isolation: The system isolates the trigger interval between the two temporal peaks P1 and P2 before extracting the control motion.The detected rolling window and identified root joint provide the starting data for this isolation.
- Motion Isolation: A kinematic subtree containing the detected root and its distal joints isolates the active limb and suppresses background task motion.For a shoulder root, the subtree includes the shoulder, elbow, and wrist.
- Pose Representation: Each joint is represented by relative rotation and position delta, concatenated across the subtree into a unified pose vector.The representation combines 6D joint features into x ∈ R6K.
- Axis Extraction: PCA on mean-centered pose vectors produces the first principal component w1, which defines the one-dimensional motion axis.The axis captures the dominant structure of the isolated trigger motion.
- Signal Gating: Two quality gates require the motion to be fundamentally one-dimensional and to span enough physical distance to reject micro-twitches.The linearity gate uses explained variance, while the range gate filters insufficient spatial span.
- Control Mapping: After both gates pass, the trigger’s spatial span sets input gain and subsequent motion along w1 maps to continuous scalar UI values.The normalized signal can be interpreted through different activation functions for continuous, integrated, or discretized control.
5 System Evaluation
DOBI was evaluated first for controlled one-dimensional motor performance and then during realistic hand-busy activities. The studies found proportional control across four body regions and reliable, highly usable dual-task interaction.
- Evaluation Design: The evaluation combined a controlled 1D Fitts’ law study with a dual-task usability study involving opportunistic body-region recruitment during physical activities.The two studies separately assess baseline motor performance and use during ongoing tasks.
- 5.1 Motor Performance: 4200 Fitts’ law trials measured Movement Time, Error Rate, Effective Width/ID, and Throughput for normalized virtual-slider control.Fourteen participants completed the within-subjects reciprocal tapping task.
- 5.1 Motor Performance: R^2 values ranged from 0.74 to 0.97 across body regions, confirming proportional 1D control consistent with standard human motor behavior.The Shoulder model was strongest at R^2 = 0.97, while the Wrist was marginally significant at R^2 = 0.74.
- 5.1 Motor Performance: 2.23 bits/s overall throughput and a 5.0% error rate were observed across the four body regions.Throughput differed significantly by input condition, with distal regions outperforming proximal ones, while Error Rate did not differ significantly.
- 5.2 Dual-Task Usability: Across 186 dual-task trigger attempts, 173 were successful, 2 were false positives, and 11 required re-triggering after unsatisfactory mappings.The study covered cooking, carrying boxes, and walking while participants operated interactive widgets.
- 5.2 Dual-Task Usability: An average SUS score of 84.2 indicated high usability during realistic hand-busy activities.Ratings were highest for responsiveness, ease of operation, and task-flow preservation, and lowest for perceived naturalness.
5.3 Discussion
DOBI supports reliable, low-effort spare-body control during ongoing activities while adapting mappings to body-region capabilities and user-selected movements. Its dynamically relocatable channel extends non-hand XR input beyond fixed modalities.
- Study implications: SUS = 84.2, with dual-task users reporting reliable, well-received interaction during realistic concurrent activity.The throughput results are baselines from novice users’ instructed control phase, excluding trigger and commit overhead.
- Study implications: Distal joints support faster, finer-grained control, whereas proximal joints better suit coarse adjustments or discrete actions.Designers can accommodate this performance gradient by tailoring activation functions without changing the core motion-to-control pipeline.
- Study implications: Individual differences in articulation style were a primary source of dual-task failures, suggesting per-user trigger calibration could reduce errors.Participants used noticeably looser or tighter triggers.
- Study implications: Dynamic recruitment lets users choose a low-effort available motion as posture, range of motion, and physical constraints change during activity.This choice shifts mental cost from a fixed mapping to a lightweight user decision rather than eliminating it.
- Broader significance: DOBI offers a dynamically relocatable within-body channel alongside gaze, voice, and foot input instead of committing to one fixed modality.The authors identify context-aware arbitration among simultaneously available non-hand channels as a direction for future research.
- Broader significance: DOBI maps normalized spare-body movements to proportional, rate-controlled, step-based, or threshold-activated UI behaviors.The four activation types support continuous zooming, timeline scrubbing, discrete navigation, and binary triggers.
6 Illustrative Application Scenarios
DOBI converts spare-body motion into continuous 1D signals that can drive multiple UI behaviors. The scenarios match activation functions and body regions to social, spatial, domestic, and physical constraints.
- Activation patterns: DOBI’s normalized 1D signals support proportional, rate-controlled, step-based, and threshold activation without changing the underlying pipeline.These mappings cover continuous zooming, timeline scrubbing, discrete navigation, and binary triggers.
- Socially subtle control: 2.62 bits/s throughput and 𝑊e = 0.082 precision make the wrist suitable for socially subtle, precise analog control.Micro-movements under a table can scroll AR notes or zoom 3D models where large gestures are conspicuous.
- Spatially constrained transit: R^2 = 0.97 linearity makes the shoulder suitable for rate-controlled timeline or volume adjustments in spatially constrained environments.A stable posture offset becomes a continuous adjustment requiring minimal physical space.
- Hands-busy domestic tasks: 2.40 bits/s throughput supports swift elbow-based step navigation for deliberate recipe-dashboard selections during messy domestic tasks.Vertical elbow movements enable discrete jumps between recipe items or timers without overshooting.
- Severe physical encumbrance: Threshold activation turns head tilts into binary call decisions when both hands are occupied carrying a heavy box.The predefined threshold enables eyes-free acceptance or decline through a discrete “swipe.”
7 Limitations and Future Work
The evaluation has limited scope: it establishes controlled 1D and initial dual-task evidence, but broader generalization, higher-dimensional control, lower-body use, and unconstrained robustness remain open.
- Scope of current evaluation: Only 6 participants across 3 scenarios were included in the dual-task study, so broader generalization requires larger, longer studies.Future work should test more users, tasks, and physical contexts.
- Elicitation analysis: The elicitation study sampled five discrete control levels per articulation, constraining linearity estimates and potentially biasing R^2 upward.Continuous trajectories require denser characterization; the Fitts’ law study provides complementary motion evidence with R^2 = 0.74–0.97.
- Richer continuous input: The current 1D formulation does not establish higher-dimensional control, which would require two separable, learnable, and stable axes.This remains an open question across users, joints, and contexts.
- Beyond the upper body: The prototype and evaluations focus on upper-body joints, leaving lower-body interaction untested under gait, balance, and locomotion interference.Future work must assess knees, ankles, and feet in standing and walking contexts.
- Trigger robustness: Lab-collected trigger data may not capture unscripted background motions, atypical articulation styles, or extended-session behavior.The authors call for longer unconstrained recordings and deployment metrics such as false activations over time and end-to-end overhead.
- Tracking hardware: The prototype depends on external Vive Ultimate Trackers fused with headset body tracking, adding setup overhead and limiting immediate deployability.The skeletal-stream pipeline could pair with less cumbersome pose-estimation methods.
8 Conclusion
DOBI recruits an available spare body region for continuous 1D XR control when users’ hands are occupied. Across elicitation and evaluation studies, body-region preferences varied by context, poses followed a dominant principal axis, and the technique supported reliable interaction.
- DOBI enables continuous 1D control by recruiting an available spare body region when users’ hands are occupied.The technique combines gaze targeting, trigger detection, kinematic subtree extraction, and runtime PCA-based mapping.
- Preferred body regions vary with context, while sampled poses within a chosen articulation are often approximated by a dominant principal axis.
- 2.62 bits/s was the maximum throughput across four representative body regions in the controlled 1D Fitts’ law evaluation.The evaluation established baseline motor performance for the paradigm.
- Distal joints supported faster, more precise control than proximal joints.
- SUS = 84.2 was reported in a dual-task usability study during realistic hand-busy activities.The study showed reliable, well-received interaction.
A Dual-Task Usability Questionnaire
The dual-task usability questionnaire measured task continuity, control quality, and contextual fit using six custom 7-point Likert items. Items were rated from 1 to 7, with means and standard deviations reported in Figure 11.
- Six custom 7-point Likert items assessed Task Continuity, Control Quality, and Contextual Fit.Task Continuity comprised Q1–Q2, Control Quality Q3–Q4, and Contextual Fit Q5–Q6.
- 1 to 7 was the response scale, ranging from strongly disagree to strongly agree.
- [activity] was replaced by the participant’s current primary task: cooking, carrying boxes, or walking.
- Per-item means and standard deviations were reported in Figure 11.
- Table 3 presents the full questionnaire text grouped by concept.