Source-linked AI summary
Preprint Touch-less Interactive Augmented Reality Game on Vision Based Wearable Device
Zhihan Lv, Alaa Halawani, Shengzhong Feng, Shafiq ur Rehman, Haibo Li
TL;DR
The paper investigates how touch-less interaction can support augmented-reality games on vision-based wearable devices. It combines gesture recognition and tracking with three games and evaluates usability, acceptability, workload, emotions, and satisfaction. The studies report intuitive, controllable interaction and generally favorable user responses, while limiting the gesture scope to hands and feet.
Problem
Augmented-reality games face occlusion, fat-finger, and limited-intuition problems with touch or one-dimensional inputs, motivating touch-less interaction on wearable devices.
Method
The paper uses CTM, skin detection, and TLD tracking to recognize dynamic hand and foot gestures and evaluates three augmented-reality games on a wearable framework and Google Glass.
Results
User studies find touch-less interaction intuitive and controllable, with Google Glass a preferable platform and favorable satisfaction and social-acceptability results.
Takeaways & Limitations
Vision-based smart-glass interaction can support touch-less augmented-reality games without relying on range sensors or dual cameras.
Takeaways & Limitations
The presented touch-less games are limited to hand and foot gestures and do not consider normal full-body gestures.
Abstract
from arXiv · showhide
This is the preprint version of our paper on Personal and Ubiquitous Computing. There is an increasing interest in creating pervasive games based on emerging interaction technologies. In order to develop touch-less, interactive and augmented reality games on vision-based wearable device, a touch-less motion interaction technology is designed and evaluated in this work. Users interact with the augmented reality games with dynamic hands/feet gestures in front of the camera, which triggers the interaction event to interact with the virtual object in the scene. Three primitive augmented reality games with eleven dynamic gestures are developed based on the proposed touch-less interaction technology as proof. At last, a comparing evaluation is proposed to demonstrate the social acceptability and usability of the touch-less approach, running on a hybrid wearable framework or with Google Glass, as well as workload assessment, user's emotions and satisfaction.
1 Introduction
The paper addresses touch and pointing limitations in augmented-reality games by investigating touch-less gestures on vision-based wearable devices. It develops games and evaluates usability, acceptability, workload, emotions, and satisfaction.
- Augmented reality blends computer-generated objects with real footage in real time, enhancing immersion while increasing interaction complexity.
- Touch-controlled augmented-reality games face occlusion and fat-finger problems, while touch-less interaction can provide finer mobile interaction resolution.
- Touch-less hand and foot gestures support more natural interaction patterns that can help players focus on gameplay rather than input.
- Wearable cameras and touch-less methods offer an interaction bridge for mobile play on the move, where desktop-oriented WIMP techniques are unsuitable.
- The study implements three augmented-reality games using real-time hand and foot detection and evaluates eleven dynamic gestures with questionnaires covering usability, workload, emotions, and satisfaction.
2 System
The system combines vision-based wearable hardware with augmented-reality game software that converts tracked hand or foot motion into touch events.
- The system has vision-based wearable hardware and touch-less interaction-based augmented-reality game software.
- Users wear the framework on their wrists or knees or smart glasses on their heads while the software tracks finger or foot motion before the camera.
- Dynamic gestures are used to simulate touch events for interaction with the games.
2.1 Gesture Recognition Algorithm
The gesture-recognition algorithm combines contour-based template matching, lightweight skin detection, and TLD tracking to support real-time hand and foot interaction on wearable devices.
- The algorithm combines dynamic-programming-based CTM localization, lightweight skin detection, and TLD tracking for hand and foot gestures.
- CTM finds the best camera-view region containing a human finger or foot contour by comparing an offline template with smartphone-camera frames.
- Dynamic programming is used to obtain a globally optimal template match for the detected contour.
2.2 Enhanced Gesture Detecting and Tracking
The enhanced pipeline combines deformable contour matching, skin-based hypotheses, TLD tracking, and image scaling to improve gesture detection and tracking efficiency on mobile hardware.
- 2.2.2 Contour-based Template Matching: CTM compares binary templates with binary edge images to locate hand or foot contours while allowing controlled template deformation.
- 2.2.2 Contour-based Template Matching: Dividing a template into movable segments permits shape changes; 3-pixel segments allow approximately 33% overall enlargement or shrinkage.
- 2.2.2 Contour-based Template Matching: The Viterbi dynamic-programming matcher searches a trellis for the path maximizing accumulated score, with node rewards based on covered edge pixels.
- 2.2.3 CTM-TLD: CTM initializes TLD because generic models cannot initialize TLD for interactive hands, and CTM supplies the region of interest.
- 2.2.3 CTM-TLD: TLD was selected over HoughTrack and PixelTrack because it initializes in less than 1 second while retaining high tracking accuracy and supporting scale variation.
- 2.2.4 Optimization: Image scaling uses 25% for foot and 12.5% for hand in the first frame, then 12.5% for foot and 6.25% for hand with TLD tracking.
- 2.2.4 Optimization: Hand-target and foot-target tests report tracking efficiency and success rates close to 100%.
2.3 TechnicalEvaluation
The proposed gesture-recognition system operates in real time and maintains high recognition accuracy across varied motion, blur, lighting, and background conditions. Evaluation also examines tracking precision, border-region recognition, and battery effects.
- Recognition accuracy: 99.76% overall gesture-recognition success was achieved across 120 mobile video sequences containing varied gesture sizes, orientations, lighting, and real environments.The sequences each contained more than 1000 frames and allowed up to 33% deformation.
- Recognition accuracy: Real-time execution above 10fps remained accurate during fast gesture movement and image blur for the hand and foot gestures used in the games.Recognition depends on detecting the correct hand or foot texture in the relevant processing step.
- Robustness: The ROI selection method can freely expand or shrink the recognition region, allowing gestures near camera borders to be recognized.This addresses difficulty detecting hands pointed toward lower or peripheral screen regions.
- Tracking precision: 1.516 pixels mean error was measured for finger motion on the XY axis across twenty users, with sd = 0.8572.The error was calculated per 15-degree motion and the maximum recorded value was 2.8 pixels at 255 degrees.
- Tracking precision: 1.6857 pixel mean error was measured during motion, with sd = 0.7255 and a maximum error of 3 pixels when finger-motion speed changed.The test evaluated tracking performance during finger motion.
- Battery performance: Touch-less gestures drain battery life by 20%, while meaningful power savings occur when the camera remains open without gestures in view.The no-gesture condition saved 12% power compared with computing gestures when hands or feet appeared in the camera view.
3 Use Cases
The use cases demonstrate touch-less hand and foot interaction across ordinary postures and wearable configurations. The system supports handheld devices, a wrist- or knee-mounted framework, and Google Glass.
- Motion Sensing Gesture: Hand and foot gestures are designed for pervasive play while users are sitting, standing, lying, or moving in ordinary life.The system uses a linear method for foot-motion sensing and can trigger events related to position in a scene.
- Motion Sensing Gesture: The gesture set includes finger swings, hand movement, finger flexion and extension, tiptoeing, and other foot motions that trigger scene events.Finger flexion and extension resembles clicking, while foot gestures can control position- or rhythm-related interactions.
- Running on Handheld Devices: Handheld devices recognize gestures through rear or front cameras across sitting, standing, and lying postures, although feet are farther from the camera.The camera capture zone and user sight are illustrated for these configurations.
- Running on Hybrid Wearable Framework: The hybrid wearable framework uses a mobile phone holder, holder base, and fastener strip, mounting on wrists for hand interaction or knees for foot interaction.An active loose leaf adjusts the smartphone camera pitch angle for wrist and knee use.
- Running on Smart Glasses: Google Glass avoids occlusion and fat-finger problems but its one-dimensional touchpad limits the intuition and flexibility of interaction.The proposed vision system instead interprets real-time camera-captured hand and foot movements without requiring users to hold a device.
- Running on Smart Glasses: On Google Glass, the camera’s capture zone largely coincides with the wearer’s sight because the eyes and camera share a close position and orientation.Gestures visible to the wearer are therefore generally also visible to the camera.
4 Proof of Concept
Three primitive augmented reality games serve as proof-of-concept applications for touch-less interaction. They cover hand-based movement, foot-based football play, and rhythm control through foot gestures.
- Proof of Concept: Three augmented reality games demonstrate the proposed touch-less interaction system across multiple use cases.The games are presented to examine the approach’s usefulness, viability, flexibility, possibilities, and limitations.
- Interaction loop: The system overlays a rendered glove, shoe, or color-marked cursor and maps physical hand or foot motion onto screen coordinates.This gives users immediate visual reference of hand or foot position relative to the screen.
- Bouncing Ball Game: Bouncing Ball Game uses speed- and orientation-based gestures to intercept, dribble, throw, or bunt an augmented reality ball on a handball court.The game renders an augmented reality glove following the user’s finger gesture.
- Football Game: Football Game uses real-foot kicking and movement gestures within a multimodal augmented reality scene containing video, audio, and vibration.Players use foot gestures to kick and dribble the ball.
- Foot-Play Piano: Foot-Play Piano demonstrates rhythm control by using foot gestures to play music on an Android smartphone.A circular progress plate indicates the duration of a pressed piano key, including whole- and half-Mora values.
5 User Study
The user study evaluated social acceptability, gesture usability, workload, emotions, and satisfaction across the proposed augmented reality games and wearable platforms. Results generally supported applicable gestures and low workload, while foot interaction produced higher workload responses.
- Study design: The study used comparative questionnaires to assess social acceptability, gesture usability, workload, emotions, and user satisfaction.Participants viewed gesture tutorials, tried the devices, played three augmented reality games, and completed multiple assessments.
- Social acceptability: 0.6500, 0.6222, and 0.6056 were the mean acceptance ratings for Bouncing Ball, Football, and Foot Piano, respectively.All games exceeded 0.5 across contexts; Football was slightly more acceptable than Foot Piano.
- Usability: All designed gestures were judged applicable, with Finger Flexion and Extension receiving the highest usability score and Swing Finger Slowly the lowest.The first user group showed significant usability differences across gestures (χ2 = 24.63, p = 0.0061).
- Usability: The second group reported higher usability means than the first for every gesture except Swing Finger Fast, with a significant overall effect (χ2 = 18.64, p = 0.0451).The comparison did not indicate that the designed gestures were exhaustively restricted by the device.
- Workload: NASA-TLX ratings were generally low, although foot interaction scored higher across workload dimensions and Foot Piano frustration was nearly twice that of hand-based Bouncing Ball.Users generally reported low mental and physical demand, with feet used somewhat more frequently than hands.
- Emotion: Pleasant emotions exceeded unpleasant emotions for both platforms, with Google Glass receiving pleasantness ratings of 3 for both hand and foot interaction.The Geneva Emotion Wheel was administered after hand- and foot-interaction tasks.
6 Discussion
The discussion reports that touch-less interaction was generally intuitive, controllable, and acceptable across the evaluated games and devices. Google Glass offered a more intuitive experience, while hand interaction was more acceptable and less demanding than foot interaction.
- Social acceptability: Participants preferred playing the three augmented reality games in a pub, whereas acceptability was low in colleague and workplace contexts.The result identifies context-dependent social acceptability rather than uniform acceptance across locations.
- Usability: Google Glass generally produced a more intuitive and comfortable user experience, while its smart-glass form did not change gesture applicability.The discussion attributes gesture differences to device fixation and available joints rather than to a general incompatibility of the gestures.
- Workload: Hand interaction received lower mental and physical demands than foot interaction, leading users to prioritize the hand for the same mission.The workload result is consistent with the observed preference for hand-based interaction.
- Emotion: Touch-less interaction produced more positive than negative emotional responses, but Google Glass did not significantly improve control over the proposed gestures.The emotional pattern was observed across both evaluated devices.
- Satisfaction: Participants rated the touch-less interfaces as interesting and comfortable, though Comfort and Willingness received comparatively lower scores.The discussion interprets the overall satisfaction results as supporting touch-less interfaces as substitutes for touch-based mobile devices.
- Platform comparison: The study concludes that smart glasses are a preferable platform because they free the hands and expand the operation space for touch-less gestures.The discussion links the platform preference to the camera and display arrangement and the availability of both hands.
7 Conclusion
The paper presents a real-time touch-less interaction technology and evaluates it through augmented reality games on vision-based wearable devices. Its conclusion favors smart glasses as a platform, while noting that the demonstrated games remain limited to hand and foot gestures.
- Contribution: The paper proposes dynamic hand- and foot-gesture interaction and evaluates it through three augmented reality games on vision-based wearable devices.The work compares a hybrid wearable framework with smart glasses.
- Conclusion: Smart glasses are identified as the preferable platform for the proposed touch-less interaction approach.The conclusion describes the approach as providing a full view of body control compared with traditional interaction technology.
- Limitation: The demonstrated games are limited to hand and foot gestures, while normal full-body gestures were not considered.This limitation defines the scope of the research version rather than a general claim about touch-less interaction.
- Future possibilities: The hybrid wearable framework could be worn on other body parts or combined with alternative sensors to improve the interaction modality.The paper presents these possibilities as potential extensions of the framework.