PaperlayerView exact revision on arXiv

Source-linked AI summary

Mobile Augmented Reality: User Interfaces, Frameworks, and Intelligence

Jacky Cao, Kit-Yung Lam, Lik-Hang Lee, Xiaoli Liu, Pan Hui, Xiang Su

arXiv:2106.08710v1cs.HCcs.AIcs.CVcs.CYcs.LG

TL;DR

The survey identifies an unresolved gap in adaptive, personalized MAR for dynamic environments and user behavior. It reviews 37 frameworks and related work, finding uneven feature coverage and limited body tracking.

  • Problem

    The survey identifies a gap in adaptive, personalized MAR that accounts for dynamic tangible environments and user behavior.

    Source: §6.4 · p. 25

  • Method

    The article uses a top-down review, selecting and comparing 37 MAR frameworks and synthesizing work on applications, adaptive visualisation, and machine learning.

    Source: §7 · p. 27

  • Results

    Across 37 frameworks, Android and iOS are most consistently supported, body tracking appears in only two, and average feature support is about three.

    Source: §4.2 · p. 18

  • Takeaways & Limitations

    Multi-user MAR frameworks face privacy challenges in sharing sensitive information, including users’ physical locations.

    Source: §6.6 · p. 26

    The authors report no definitive solutions yet for adapting MAR to dynamic environments and user behavior.

    Source: §6.4 · p. 25

Abstract

from arXiv · show

Mobile Augmented Reality (MAR) integrates computer-generated virtual objects with physical environments for mobile devices. MAR systems enable users to interact with MAR devices, such as smartphones and head-worn wearables, and performs seamless transitions from the physical world to a mixed world with digital entities. These MAR systems support user experiences by using MAR devices to provide universal accessibility to digital contents. Over the past 20 years, a number of MAR systems have been developed, however, the studies and design of MAR frameworks have not yet been systematically reviewed from the perspective of user-centric design. This article presents the first effort of surveying existing MAR frameworks (count: 37) and further discusses the latest studies on MAR through a top-down approach: 1) MAR applications; 2) MAR visualisation techniques adaptive to user mobility and contexts; 3) systematic evaluation of MAR frameworks including supported platforms and corresponding features such as tracking, feature extraction plus sensing capabilities; and 4) underlying machine learning approaches supporting intelligent operations within MAR systems. Finally, we summarise the development of emerging research fields, current state-of-the-art, and discuss the important open challenges and possible theoretical and technical directions. This survey aims to benefit both researchers and MAR system developers alike.

1 INTRODUCTION

MAR has developed across mobile devices, but user-oriented design of its frameworks remains an open concern. This survey connects applications, adaptive interfaces, frameworks, and machine learning with a research agenda for more intelligent, user-oriented MAR.

1 INTRODUCTION

  • The survey covers MAR applications, interface designs, frameworks and SDKs, supported hardware and features, and machine-learning methods for key components.It also outlines challenges and research opportunities for user-oriented interfaces and supporting AI methods.Source: §1 · p. 2
  • The survey’s agenda targets adaptive user interactions, ML for dynamic environments, and two-way engagement with intelligent objects.The authors describe these as future research directions rather than established solutions.Source: §1 · p. 3

1.1 Overview of Mobile Augmented Reality

  • A typical MAR pipeline captures sensor data, analyzes it through machine-learning tasks, then returns results for tracking and adaptive rendering.Tasks may run on the device or an external server.Source: §1.1 · p. 3, Figure 1 · p. 4

1.2 Related Surveys and Selection of Articles

  • Earlier surveys examined particular AR areas, while this article addresses gaps with a combined review of interfaces, frameworks and SDKs, and ML methods.Its article set was built through keyword searches, citation expansion, media additions, and relevance screening.Source: §1.2 · pp. 3–4

1.3 Structure of the survey

  • The article proceeds from MAR applications to adaptive UIs, frameworks, ML, research challenges, and conclusions.Source: §1.3 · p. 4

2 MAR APPLICATIONS

MAR applications span entertainment and tourism, education, and healthcare, with interface needs shaped by each setting. Examples range from traveller-tailored guidance and classroom support to medical imaging and surgical guidance.

2.1 Entertainment & Tourism

  • Entertainment and tourism applications include AR games and social-media effects, as well as traveller-tailored guides with multilingual, haptic, or audio options.Tourism examples include city navigation and sightseeing information tailored to visitors.Source: §2.1 · p. 5

2.2 Education

  • Educational AR includes subject-specific learning applications and teaching tools designed to support interactive content or communication.Reported benefits include improved information handling, confidence, and spatial abilities; applications also address individual learning needs and teacher-student communication.Source: §2.2 · pp. 5–6

2.3 Healthcare

  • Healthcare uses include assistance for patients and clinicians, 3D medical-image visualisation, surgical planning, image guidance, and training.In image-guided surgery, tracked instruments are visualised against patient-specific datasets; UI techniques also address occlusion and registration error.Source: §2.3 · p. 6

3 COMPOSING AR VISUALISATION AND USER ENVIRONMENTS

The section surveys adaptive MAR interfaces, collaborative multi-user and multi-device designs, and approaches to evaluating AR user experience. It identifies context-sensitive presentation and systematic evaluation as continuing design concerns.

3.1 Adaptive UI for MAR

  • Adaptive UI work spans filtering and clustering to reduce clutter, alongside depth cues and illumination estimates that support depth perception and visual coherence.The survey groups adaptive techniques around filtering, mapping, and rendering.Source: §3.1 · pp. 7–8
  • Registration and sensing errors can cause sensory conflicts and misalignment, so interfaces should adapt dynamically and place labels relative to visible object parts.Error-adaptive visualisations can add context or show relationships between an augmentation and nearby physical objects as registration error changes.Source: §3.1 · pp. 7–9

3.2 Collaborative UIs in Multi-user and Multi-device AR

  • Collaborative AR must manage awareness, control, information transparency, privacy, and interaction across diverse devices; many evaluations neglect users’ responses to physical environments.The section notes that existing evaluations often rely on a small set of quantitative metrics such as completion time and error rate.Source: §3.2 · p. 10

3.3 Metric Evaluations for AR UIs

  • User-centric evaluation extends beyond work outcomes to perception, usability, task success, subjective experience, and handheld-device ergonomics.AR interface plasticity concerns compatibility between information, physical surroundings, and users’ situations.Source: §3.3 · pp. 10–11
  • Interactive evaluation setups can be costly and time-consuming, while existing AR UI evaluation frameworks are few and context-specific.The section points to systematic or automated evaluation as an opportunity, including real-time performance monitoring and information collection through user-device interaction.Source: §3.3 · p. 11

4 MAR FRAMEWORKS

The survey compares MAR frameworks by platform support, tracking, features, sensors, and processing architecture. Across 37 frameworks, support is uneven across tracking and features, while newer capabilities may constrain backward compatibility.

4 MAR FRAMEWORKS

  • As frameworks add features, older devices may lack the processing capabilities or hardware required, leading to omitted features or discontinued device support.The survey notes that software optimisation can support a variety of mobile hardware, but backward compatibility can decrease as feature requirements grow.Source: §4 · pp. 11–12

4.1 Evaluation Metrics

  • The comparison evaluates platform support, tracking technologies, features, sensors, and whether processing is offline, online, or both.Tracking includes marker-based and markerless approaches, as well as device, plane, hand, body, and facial tracking.Source: §4.1 · pp. 12–13

4.2 Frameworks

  • 30 of 37 frameworks support 2D image marker tracking, while only 2 support 2D and 3D body tracking; 28 support offline-based or combined offline and online processing.The table compares framework and SDK features; the survey reports an average of approximately 3 supported features per framework.Source: Table 1 · p. 14, §4.2 · p. 18

5 MACHINE LEARNING METHODS FOR MAR

This section surveys machine-learning methods used across MAR tasks, from object detection and tracking to adaptive interfaces. It also compares deployment patterns and describes methods for responding to users and changing environments.

5 MACHINE LEARNING METHODS FOR MAR

  • The survey organizes ML work by AR task and deployment location, then maps task–method links, with thicker arrows indicating more associated works.The examples distinguish server, edge, and client deployment.Source: §5 · pp. 18–19, Table 2 · p. 19, Figure 3 · p. 21

5.1 ML for Object Detection and Object Recognition

  • Object detection and recognition use deep-learning models, commonly CNN families, deployed on servers, edge devices, or clients depending on available resources.Efficient networks and mobile libraries support client deployment despite device computation and battery limits.Source: §5.1 · pp. 19–20

5.2 ML for Object Tracking

  • ML-based tracking is less common than detection and recognition; MAR systems often combine learned detection with conventional tracking, while exploratory RNN work models temporal dependencies.The passage attributes the scarcity of deep-learning trackers partly to a lack of suitable tracking training data.Source: §5.2 · pp. 21–22

5.3 ML for Adaptive UIs

  • For adaptive interfaces, ML supports information filtering, clustering, layout generation, and context-aware recommendations, although one described recommendation study reports no performance results.ReadMe is reported to outperform distance-based and focus-based recommendation baselines.Source: §5.3 · p. 22
  • In ex vivo liver AR surgery, the physics-based DNN registers in 3 ms with a mean TRE of 2.9 mm and is 500× faster than a reference finite element solution.TRE values are used to highlight regions around the virtual liver model as an error warning.Source: §5.3 · pp. 22–23
  • Adaptive-UI methods also use ML for occlusion and depth cues, hand segmentation for virtual-object interaction, and illumination estimation for more accurate light-direction estimates.DeepLight is reported to estimate light direction more accurately than the Gardner et al. approach.Source: §5.3 · p. 23

6 RESEARCH CHALLENGES AND FURTHER DIRECTIONS

The survey identifies open challenges for MAR spanning immersive interaction, audio, user workload, changing environments, situational awareness, privacy, and federated learning. It outlines design directions while noting unresolved gaps and practical trade-offs.

6.1 Towards seamless frameworks for interactable immersive environments

  • Next-generation frameworks face an open challenge in merging digital content with physical surroundings and designing spatial environments for interaction among users, virtual entities, and tangible objects.The survey also raises how to evaluate AI-supported animated objects interacting with people and physical objects.Source: §6.1 · p. 24

6.2 Audio augmented reality

  • Audio interaction remains neglected in frameworks; proposed design considerations include individualization, context awareness, and audio that reflects users’ social context.The survey points to aligning audio with the appearance and context of virtual visuals as a research topic.Source: §6.2 · p. 24

6.3 User interaction and user workload

  • Few frameworks support on-body interfaces, and none integrate workload measurements; the survey calls for interaction traces and systematic evaluation to improve interface design.The discussion covers tasks ranging from selecting a 2D menu button to manipulating a 3D object.Source: §6.3 · pp. 24–25

6.4 Users and their environments

  • Changing tangible environments and unanticipated user behavior can leave AR information outdated or misaligned, motivating adaptive, personalized information management for which the survey offers no definitive solution.Repeated use of an unofficial route can change the environment while AR guidance lags behind.Source: §6.4 · p. 25

6.5 Deepening the understanding of users’ situations

  • GPS-based tracking precision is moderate in many applications, while camera-based detection can improve precision at the cost of overhead, network traffic, and latency from distant cloud servers.The survey identifies client-server architecture as a design challenge under these trade-offs.Source: §6.5 · p. 26

6.6 Privacy preserving MAR frameworks

  • Multi-user collaboration is rarely addressed because sharing spatial, policy, and interaction-trace information can create privacy risks, including conflicts among users’ privacy preferences.The survey calls for privacy protections at both application and multi-user framework levels.Source: §6.6 · p. 26

6.7 Federated learning for MAR

  • Federated learning can locally update models and send updates for server aggregation, supporting personalized models while reducing communication latency; viewport prediction is one proposed MAR use.The described approach trains a global model in distributed fashion while retaining a personal model for fine-grained personalization.Source: §6.7 · pp. 26–27

7 CONCLUSIONS

The survey reviews MAR applications, visualizations, frameworks, and machine-learning methods, compares 37 frameworks and SDKs, and identifies challenges and unexplored topics for user-centric systems.

7 CONCLUSIONS

  • The survey reviews MAR applications, adaptive visualizations, 37 frameworks and SDKs, and ML methods, then discusses challenges and unexplored topics for seamless, user-centric frameworks.Its top-down review compares framework functions and features and examines how ML supports sensitivity to user contexts and physical environments.Source: §7 · p. 27

Related papers

Loading 2106.08710v1…