Source-linked AI summary
Interaction Methods for Smart Glasses
Lik-Hang Lee, Pan Hui
TL;DR
Smart glasses still face difficult human-computer interaction because their small, non-touchable displays and constrained controls make direct manipulation problematic. This survey examines market devices and research on handheld, touch, and touchless input, emphasizing on-device, on-body, hands-free, and freehand methods. It classifies these approaches, compares them across eight interaction goals, and discusses design challenges and multimodal input, including evidence that optimized PalmType text entry reached 10 words per minute.
Problem
Smart glasses have small displays and constrained input, making direct manipulation fatiguing and error-prone while limiting effective augmented-reality interaction.
Method
The survey studies market devices and literature, classifying interaction into handheld, touch, and touchless input, with touch divided into on-device and on-body and touchless into hands-free and freehand.
Results
The survey compares four touch and touchless categories across eight interaction goals and summarizes existing research efforts, trends, and interaction challenges.
Takeaways & Limitations
The surveyed methods support a classification framework for smart-glass interaction and point to multimodal input as an opportunity for hybrid augmented-reality interfaces.
Takeaways & Limitations
External interaction devices add setup burden, while small on-device surfaces can cause muscle fatigue and require subtle finger movements that degrade task performance.
Abstract
from arXiv · showhide
Since the launch of Google Glass in 2014, smart glasses have mainly been designed to support micro-interactions. The ultimate goal for them to become an augmented reality interface has not yet been attained due to an encumbrance of controls. Augmented reality involves superimposing interactive computer graphics images onto physical objects in the real world. This survey reviews current research issues in the area of human computer interaction for smart glasses. The survey first studies the smart glasses available in the market and afterwards investigates the interaction methods proposed in the wide body of literature. The interaction methods can be classified into hand-held, touch, and touchless input. This paper mainly focuses on the touch and touchless input. Touch input can be further divided into on-device and on-body, while touchless input can be classified into hands-free and freehand. Next, we summarize the existing research efforts and trends, in which touch and touchless input are evaluated by a total of eight interaction goals. Finally, we discuss several key design challenges and the possibility of multi-modal input for smart glasses.
1 INTRODUCTION
Smart glasses face usability and interaction challenges because their small displays and limited input controls make direct manipulation difficult. This survey reviews market devices, classifies touch and touchless techniques, and compares them across interaction goals while discussing multimodal opportunities and design challenges.
- Smart glasses overlay virtual content on a see-through display, but their interaction remains encumbered by small displays, limited interfaces, computational constraints, and short battery life.
- Google Glass uses swipe gestures and a Timeline of pixel cards, but this design limits micro-interactions and can increase search time as card numbers grow.
- Existing touch pads and buttons can produce long task times and high item-selection error rates, while no standard mature interaction method has emerged.
- The survey reviews smart glasses, their sensors, and interaction methods, then examines touch-based approaches including on-device and on-body techniques.
- It investigates touchless input through external devices and sensors, focusing on hands-free and freehand interaction techniques.
- The survey organizes touch and touchless inputs into four categories, compares them across eight interaction goals, and discusses hybrid interfaces and multimodal input.
2 PRELIMINARY- THE INTRODUCTION OF SMART GLASSES AND THEIR SENSORS
Smart glasses combine see-through displays, sensors, processing, and input methods to overlay digital information on the physical world. Their development spans bulky early prototypes and specialized systems to lightweight commercial devices, while usability remains constrained by hardware and interaction limitations.
- Overview: Smart glasses use optical head-mounted displays, sensors, and processing capabilities to present augmented information over the physical world.These capabilities support interaction between users and augmented-reality environments.
- Examples of smart glasses: The Touring Machine established an early GPS-driven augmented-reality system using a see-through display, orientation detector, stylus, trackpad, and handheld computer.It supported campus navigation through overlaid building information and virtual menus, despite its cumbersome hardware.
- Examples of smart glasses: Weavy reduced the form factor to a lightweight head-worn wearable but offloaded camera-based computer-vision processing to a backend server.Its limited onboard computing reflected the technological constraints of its period.
- Examples of smart glasses: WUV and BrainyHand projected augmented information onto a hand or nearby surface and relied primarily on hand gestures because their miniature size prevented trackpads or buttons.WUV used colored hand markers, whereas BrainyHand recognized simple gestures from the distance between the hand and camera.
- Challenges: Current smart glasses remain constrained by weak processors, short battery life, small screens, and unsettled input methods, making specialized task-oriented uses more plausible than general daily adoption.The survey also identifies usability and interaction design as central challenges for widespread deployment.
- Sensors and input methods: Buttons and trackpads are common but can cause muscle fatigue and impaired performance, whereas accelerometers help distinguish intentional head gestures from motion such as walking.Among 12 smart glasses, 6 provide buttons or trackpads, 3 provide external controllers, and 3 rely on camera-supported gestural input.
3 INTERACTION APPROACHES FOR SMART GLASSES
Smart-glasses interaction approaches are organized into handheld, touch, and touchless input, with touch further divided into on-device and on-body methods. The survey compares these methods across body surfaces, devices, gestures, and sensing constraints.
- Interaction approaches are classified as handheld, touch, and touchless; touch includes non-handheld body-surface input.The survey focuses mainly on touch and touchless interaction methods.
- On-device interaction: External devices such as rings, wristbands, sleeves, and belts provide precise, responsive input but add device-wearing burden.Their spatial mapping supports accurate input and fast repetition, while putting on the device is a drawback.
- On-body interaction: On-body interaction can provide tactile feedback, eyes-free input, and higher performance than touchless mid-air input.Tactile cues help users locate touches without relying on visual clues.
- On-body interaction: On-body designs vary by surface: forearms are preferred for comfort, cheeks avoid obstructing vision, and ears support only limited touch-area divisions.Thumb-to-finger interaction favors the first and second phalanges, while facial and ear input face fatigue, acceptance, or spatial constraints.
- On-body interaction: PalmType’s optimized palm keyboard reached 10 words per minute, 41% faster than a touchpad and 29% faster than its squared layout.The comparison suggests that mapping the virtual interface to palm shape influences text-entry performance.
- Touchless interaction: Freehand and gaze-based input remain constrained by speed, recognition, calibration, hardware, or interface-complexity limitations.Hand gestures can be inaccurate and fatiguing for repetitive tasks; gaze input is error-prone and requires calibration, while some tongue interfaces demonstrate only simple gestures.
4 EXISTING RESEARCH EFFORTS AND TREND
Existing research covers touch and touchless interaction methods across distinct interaction goals. Touch input is mainly used for 2D interfaces and text entry, while freehand interaction supports intuitive manipulation of virtual 3D objects and physical environments.
- More than 30 research works are compared across interaction approaches including spectacle frames, rings, wristbands, belts, body surfaces, body movements, gloves, and cameras.The comparison is organized around interaction goals and the proposed classification of touch and touchless input.
- Touch input: Touch input mainly supports tap, swipe, and text-entry interactions, providing alternative on-device and on-body approaches for smart glasses.The surveyed touch techniques target smartphone-like 2D tasks and text input.
- Touchless input: Hands-free input such as head and gaze movement is primarily limited to micro-interactions, whereas freehand input supports item selection, mid-air text entry, and 3D manipulation.Freehand methods use gloves, EMG wristbands, or vision-based sensing, depending on the interaction task.
- Touchless input: Vision-based freehand interaction is distinctive for directly manipulating virtual 3D objects and physical environments, but text entry remains a concern.The survey characterizes freehand interaction as natural and intuitive for 3D manipulation while noting that it is not an all-rounded approach.
- Multi-modal input: Multi-modal input combines touch and touchless techniques to use their complementary strengths, such as precise small-item interaction and large-item freehand selection.One surveyed design assigns hand gestures to locate large items and a finger-worn device to reinforce precise movement on small 2D items.
5 INTERACTION CHALLENGES ON SMART GLASSES
The survey identifies incomplete coverage of hybrid 2D/3D interfaces as a central challenge for smart glasses. It discusses complementary input modalities, device form factors, timing, social acceptance, and energy consumption as design considerations.
- Hybrid user interface on smart glasses: No existing work covers all eight interaction goals for hybrid interfaces containing virtual 2D and 3D contents.Virtual 2D contents are associated with TAP, TRA, MFT, KEY, and GUT, while virtual 3D contents are associated with GES, DMO, and PHY.
- Towards higher coverage of interaction goals: Touch input mainly supports fast, accurate 2D interaction and text entry, whereas touchless input dominates virtual 3D interaction.The survey attributes this division to finger dexterity for rapid 2D tasks and the naturalness of hand gestures for 3D objects.
- Building all-rounded interaction approaches: Combining touch and touchless input is proposed as a route toward broader coverage of interaction goals.The survey identifies complementary gaps: touch can improve intuitive 3D interaction, while touchless input must address tasks requiring fast repetition.
- Building all-rounded interaction approaches: Companion devices and multi-modal designs may combine large-area or precise touch interaction with freehand interaction in augmented reality.Examples include finger-worn touch interfaces paired with vision-based freehand input and haptic gloves supporting both touch and freehand interaction.
- Form size for wide-ranging coverage: The size and location of touch-sensitive areas influence how comprehensively an input design covers interaction goals.Forearms, palms, and wristbands are described as large interaction areas that can support varied tasks.
- Considering temporal factor in interaction design: Multi-modal systems must account for the temporal relationship between input modalities, including order, succession, intersection, inclusion, and simultaneity.The survey presents these temporal possibilities as a framework for designing modality switching.
- Social acceptance and appealing design: Social acceptance favors unobtrusive touch-based input over speech recognition, which is described as disturbing and obtrusive in public settings.The survey also notes that aesthetically pleasing input devices can support social acceptance.
- Energy consumption on smart glasses: Limited battery life makes energy consumption a fundamental design factor, with externally powered inputs preferred over vision-based approaches.The passage contrasts external devices or separate energy provision with vision-based input, whose energy use is a concern.
6 CONCLUSIONS
The survey organizes smart-glasses interaction research around touch and touchless input, and identifies multimodal input as a promising direction for reducing interaction encumbrance. Combining 3D hand gestures with touch-based gestures may support augmented-reality interfaces containing both 2D and 3D objects.
- The survey classifies smart-glasses interaction methods into on-device touch, on-body touch, hands-free input, and freehand input.
- It groups these methods under the broader categories of touch input and touchless input.
- Multimodal input combining different approaches is proposed as a direction for overcoming encumbered interactions with miniature smart glasses.
- Both 3D natural hand gestures and touch-based gestures are identified as important for interfaces containing 2D and 3D objects.