Source-linked AI summary
A Review on Industrial Augmented Reality Systems for the Industry 4.0 Shipyard
Paula Fraga-Lamas, Tiago M Fernandez-Carames, Oscar Blanco-Novoa, Miguel Vilar-Montesinos
TL;DR
Shipyards pursuing Industry 4.0 need practical IAR systems to support operators and shipbuilding processes. The paper reviews IAR applications and technologies, compares hardware and software options, and proposes a Cloudlet- and Fog Computing-based architecture. It concludes that software options are numerous, while hardware is not yet ready for massive deployment and Edge resources should combine fog services with Cloudlets for demanding rendering.
Problem
Shipyards need practical IAR systems and suitable technologies to support operators and processes during their transition toward Industry 4.0.
Method
The paper reviews IAR concepts, industrial and shipbuilding applications, use cases, hardware, software, and communications architectures.
Results
The review finds many software options but hardware is not yet ready for massive deployment, and proposes a three-layer Edge Computing architecture based on Cloudlets and Fog Computing.
Takeaways & Limitations
Fog Computing can support distributed, low-latency, QoS-aware IAR services, while Cloudlets are necessary for real-time rendering or compute-intensive services.
Abstract
from arXiv · showhide
Shipbuilding companies are upgrading their inner workings in order to create Shipyards 4.0, where the principles of Industry 4.0 are paving the way to further digitalized and optimized processes in an integrated network. Among the different Industry 4.0 technologies, this article focuses on Augmented Reality, whose application in the industrial field has led to the concept of Industrial Augmented Reality (IAR). This article first describes the basics of IAR and then carries out a thorough analysis of the latest IAR systems for industrial and shipbuilding applications. Then, in order to build a practical IAR system for shipyard workers, the main hardware and software solutions are compared. Finally, as a conclusion after reviewing all the aspects related to IAR for shipbuilding, it is proposed an IAR system architecture that combines Cloudlets and Fog Computing, which reduce latency response and accelerate rendering tasks while offloading compute intensive tasks from the Cloud.
I. INTRODUCTION
The paper frames IAR as a technology for supporting shipyard operators as companies pursue Shipyards 4.0. It reviews industrial and shipbuilding IAR systems, compares enabling technologies, and proposes an Edge Computing architecture.
- Shipbuilding companies are applying Industry 4.0 principles to create more digitalized and optimized Shipyards 4.0.
- IAR can support operators through task assistance, data visualization, human-machine interaction, localization, maintenance, quality control, and material management.
- The paper analyzes the latest IAR research and technologies for building systems intended for shipyards.
- It reviews recent IAR systems and compares hardware and software technologies for industrial and shipbuilding applications.
- The paper identifies shipyard use cases and proposes an IAR architecture using Edge Computing to reduce latency response and accelerate rendering tasks.
B. ESSENTIAL PARTS OF AN AR SYSTEM
An AR system captures and processes the physical environment, estimates device position, renders aligned virtual information, and presents it through a display. Its performance depends on sensing, tracking, registration, rendering, and real-time processing.
- An AR system combines image capture, processing, activation elements, and a display for presenting virtual information with the physical environment.
- Displays include video-mixed systems, which digitally merge camera and virtual information, and optical see-through systems, which optically superimpose virtual content.
- AR hardware can use hand-held, spatial, or head-mounted displays, supporting individual viewing or collaborative work.
- The AR pipeline captures a frame, estimates camera position using visual references and sensors, renders perspective-correct content, and displays it to the user.
- Interaction Handling manages image interaction, while Information Management retrieves required local or remote information.
- Key challenges include object registration, visual tracking, information display, rendering, and real-time data processing for rapid and consistent augmentation.
- Registration and tracking techniques use sensors, image-based markers, or sensor-image fusion, while Natural Feature Tracking extracts characteristic image points.
III. IAR FOR INDUSTRIAL APPLICATIONS
IAR supports diverse Industry 4.0 activities by connecting workers with contextual information, guidance, monitoring, and collaborative tools. Successful applications should add value while minimizing operational, cognitive, physical, and ergonomic burdens.
- Industrial applications: IAR supports maintenance, repair, and control through ubiquitously rendered textual, visual, or auditory instructions.It can also provide real-time information from databases, manuals, drawings, and 3D models for decision-making.
- Industrial applications: Remote assistance and augmented communication connect operators with experts and support collaborative visualization across locations.Operators can share their point of view while overlaying information on the real scene and recording annotations or audio/video.
- Industrial applications: IAR supports training, knowledge preservation, design, manufacturing, assembly, quality control, and logistics.Applications include step-by-step training, CAD evaluation, manufacturing guidance, quality monitoring, assembly simulation, and indoor warehouse routing.
- Design requirements: Successful IAR applications should provide added-value services and avoid functional, cognitive, physical, ergonomic, and interaction discontinuities.The review highlights risks including headaches, nausea, reduced visual acuity, distractions, surprises, and shocks.
IV. IAR FOR SHIPBUILDING
Shipbuilding IAR systems address welding, painting, maintenance, design interpretation, and production alignment. Existing work spans experimental systems and commercial solutions, with interfaces ranging from helmets and glasses to tablets and robot-mounted projections.
- Welding: IAR systems support shipyard welding through helmet displays, robot-mounted projections, wireless control, and virtual assistants.These systems provide task information, suggest corrections, and reduce distractions from traditional screens.
- Welding: AR welding training combines a torch, AR glasses, motion tracking, and speakers while a neural network estimates weld quality and shape.The simulation evaluates the torch’s speed and orientation in real time.
- Training: A spray-painting training system uses a force-feedback, sound-emitting paint gun and displays immediate results on virtual steel structures.Students practice painting simulated working environments shown through AR glasses.
- Maintenance and communication: Tablet-based IAR systems can display geo-referenced production notes and step-by-step maintenance instructions to support communication and reduce human error.A central element processes data from multiple sources, while tablet applications retrieve and display information.
- Design and production: An IAR tablet application visualizes, modifies, and verifies pipe segments, saves corrected geometry for bending machines, and uses optical measurement for alignment.The system addresses discrepancies between construction data and the actual ship during overlapping production and construction.
- Commercial developments: Commercial naval AR solutions have been marketed by Newport News Shipbuilding, Index AR Solutions, and BAE Systems.
V. IAR USE CASES FOR A SHIPYARD 4.0
The proposed shipyard use cases apply IAR to quality, manufacturing, localization, warehousing, maintenance, communication, hidden installations, and remote operation. Together they target more informed and efficient work across workshop and ship processes.
- Quality control: Quality control can use 3D cameras, reconstruction software, and computer vision to detect deviations from CAD models.
- Manufacturing assistance: Manufacturing assistance overlays 3D models on tangible workbench interfaces using visual markers as spatial references.
- Localization: RFID-based localization displays the 2D positions of products and tools on tablets or IAR glasses inside Navantia’s Ferrol workshop.Sensor values and artificial tags determine the user’s position in the shipyard.
- Warehouse management: Warehouse management assists storage, localization, relocation, and collection of parts while decreasing human errors and process time.
- Predictive maintenance: Predictive maintenance combines process-quality and machine or workshop sensor data, analyzes it with data-mining techniques, and presents results through IAR devices.
- Augmented communication: Augmented communication enables on-site guidance by sharing an operator’s point of view and overlaying information on the observed scene.Portable IAR devices can also record annotations and audio/video communications.
- Hidden installations: Visualization of hidden installations overlays 3D virtual elements on bulkheads, roofs, or ceilings to assist assembly, maintenance, and fault repair.Operators can inspect invisible installations after failures and decide repair operations more quickly.
- Remote operation: Remote operation superimposes a virtual control panel on smart products for headset, gesture, voice, or other-device control.
A. HARDWARE DEVICES
IAR hardware ranges from handheld devices to hands-free HMDs and spatial displays, with selection constrained by usability, environmental conditions, sensing, computation, and power. HMDs support mobile, personalized interaction, but shipyard deployment remains limited by dynamic environments, lighting, interference, and experimental hardware.
- Display and interaction devices: HMDs provide hands-free access to information, avoiding the attention and hand-use problems of tablets and smartphones during industrial tasks.Spatial Augmented Reality enables collaboration, but covering a large shipyard is difficult and expensive compared with carrying an HMD.
- Display and interaction devices: Smart-glass capabilities vary across models, including optical see-through displays, camera quality, thermal and depth sensing, microphones, and button, gesture, or voice navigation.The Daqri smart helmet includes thermal and Intel RealSense cameras, while some models provide microphones and headsets.
- Usability requirements: 30° horizontal field of view is the recommended minimum, while all-day use favors lightweight devices, full-day batteries, and low-delay optical or retinal projection.Voice interaction can free the operator’s hands but remains challenging in noisy industrial environments.
- Shipyard constraints: Shipyard deployment is constrained by limited memory, lighting-sensitive vision algorithms, compute-intensive detection, electrical interference, unreliable indoor positioning, and changing workshop geometry.Most current IAR hardware devices are still considered experimental, complicating framework integration and new feature implementation.
- Shipyard constraints: Most current IAR hardware devices remain experimental developments, making integration with existing IAR frameworks and implementation of new features difficult.
B. SOFTWARE DEVELOPMENT TOOLS FOR IAR
IAR software development requires SDK support for graphics, recognition, speech or gestures, environment reconstruction, and virtual overlays. SDK selection also depends on licensing, platforms, markers, tracking, Unity, geolocation, SLAM, cloud services, and HMD compatibility, while the ecosystem changes rapidly.
- SDK capabilities: IAR SDKs provide hardware interaction and contextual visualization through graphics, recognition, speech, gestures, 3D reconstruction, and virtual-element overlay.
- SDK selection criteria: SDK comparisons consider license type, supported platforms, marker generation, tracking, and overlay capabilities.The reviewed licensing categories include open-source, free, and commercial versions.
- SDK selection criteria: The chosen SDK should be compatible with the selected HMD device, such as Vuforia’s compatibility with Epson Moverio, ODG R-7, and HoloLens.
- SDK selection criteria: Unity compatibility, geolocation, and SLAM are important for deployment, especially for location-based applications and indoor navigation.Examples cited include UART, Vuforia, ARKit, ARCore, and Instant Reality.
- Ecosystem constraints: The SDK landscape is unstable: tools available in December 2017 could disappear, change licensing, or be replaced as commercial priorities shift.ARToolkit had recently been acquired by Daqri at the time of writing.
A. TRADITIONAL COMMUNICATIONS ARCHITECTURE
The traditional IAR architecture uses data acquisition, data transport, and visualization and interaction layers. Its cloud-centered design faces shipyard communication barriers and latency-sensitive rendering requirements, while proxy caching adds synchronization, computation, battery, and prediction costs.
- Architecture layers: The traditional IAR architecture has three layers for data acquisition, data transport, and visualization and interaction.
- Architecture layers: The VIS layer combines HMD, handheld, and spatial-display devices with interaction interfaces used by operators in ships, workshops, and shipyards.
- Architecture layers: The DTS collects DAS information and transmits it from cloud infrastructure to operators, but structural barriers, metal parts, and electrical interference hinder wireless communication.Wi-Fi can use existing IEEE 802.11 b/g/n/ac infrastructure, although propagation inside ships remains challenging.
- Architecture layers: The DAS, hosted in the shipyard cloud, can obtain work-order and installation-element data from the MES.
- Cloud-centered limitations: Dynamic IAR displays require fast loading and rendering, motivating local content storage because cloud-centered architectures can limit responsiveness.
- Cloud-centered limitations: Proxy caching introduces mobile-device computation and battery costs, synchronization complexity, and resource waste when predictions trigger unnecessary file exchanges.The proxy server caches dynamically required IAR information and can download predicted content in the background.
B. ADVANCED COMMUNICATIONS ARCHITECTURES
Advanced IAR communications architectures move processing closer to devices through local PCs, Fog Computing, Cloudlets, or Mobile Edge Computing. These approaches target lower latency and better rendering, but HMD systems still require faster interfaces and smaller network latencies.
- Edge alternatives: Local high-end PCs can store content and process video information to improve latency response and rendering performance.
- Edge alternatives: Fog Computing, Cloudlets, and Mobile Edge Computing move cloud processing power toward IAR devices at the network edge.Fog nodes may be distributed, Cloudlets use smaller cloud-like servers near users, and Mobile Edge nodes reside at cellular base stations.
- Reported architecture: An IAR Edge Computing architecture offloads demanding real-time algorithms to a high-end PC, decreasing end-to-end latency for video transmissions.
- Reported architecture: The reported Edge Computing system works for handheld devices, but most HMD solutions require higher-bit-rate communication interfaces and smaller network latencies.The authors therefore identify further study as necessary for IAR Edge Computing.
C. PROPOSED ARCHITECTURE FOR FUTURE SHIPBUILDING IAR APPLICATIONS
The proposed shipyard IAR architecture combines Fog Computing and Cloudlets in a three-layer Edge Computing design to reduce latency and rendering time. It connects distributed IAR devices and shipyard systems while addressing security and current hardware-readiness constraints.
- The architecture combines Fog Computing nodes and Cloudlets to reduce latency response and on-line rendering time.Fog nodes support distributed, low-latency applications, while Cloudlets offload rendering and reduce latency relative to the Cloud.
- The three-layer design comprises Node, Edge Computing, and Cloud layers.The Node Layer contains IAR devices; the Edge layer provides intermediate processing; and the Cloud receives, processes, and stores edge data.
- The architecture exchanges data with the shipyard’s IIoT ecosystem, including sensors, actuators, robots, and additive manufacturing devices.The Cloud can also provide access to enterprise, CAD, PLM, and IIoT services.
- Cloudlets handle sophisticated 3D CAD rendering and other compute-intensive tasks that resource-constrained devices or fog gateways cannot perform rapidly.The resulting rendered image is delivered to the IAR device faster than from the Cloud.
- Security requires further study in industrial environments, and IAR hardware is not yet ready for the massive deployment required by an Industry 4.0 shipyard.The review finds many software options, but hardware readiness remains a deployment boundary.