PaperlayerView exact revision on arXiv

Source-linked AI summary

Survey of Deep Learning for Autonomous Surface Vehicles in the Marine Environment

Yuanyuan Qiao, Jiaxin Yin, Wei Wang, Fábio Duarte, Jie Yang, Carlo Ratti

arXiv:2210.08487v3cs.ROcs.AI

TL;DR

Existing surveys have not comprehensively addressed how deep learning is implemented for autonomous surface vehicles or its strengths, weaknesses, and promising maritime applications. This paper systematically surveys DL methods across ASV navigation, guidance, control, and cooperative operations, synthesizing current applications, challenges, and future research directions. It concludes that DL has been applied across these areas while important limitations remain, including domain mismatch and handcrafted reward functions.

  • Problem

    Existing surveys lack comprehensive discussion of deep learning implementation for ASVs, including its strengths, weaknesses, applications, and promising future directions.

  • Method

    The paper systematically reviews and compares DL applications in ASVs across navigation, guidance, control, cooperative operations, and related maritime technologies.

  • Results

    The survey presents DL-based ASV applications spanning NGC systems and cooperative maritime operations, alongside current technical challenges and future research directions.

  • Takeaways & Limitations

    The review identifies DL as a framework applied to improve ASV intelligence and autonomy across maritime perception, prediction, control, and operations.

  • Takeaways & Limitations

    DL applications remain constrained by domain mismatch that introduces bias and by handcrafted reward functions that may not adapt to complex dynamic systems.

Abstract

from arXiv · show

Within the next several years, there will be a high level of autonomous technology that will be available for widespread use, which will reduce labor costs, increase safety, save energy, enable difficult unmanned tasks in harsh environments, and eliminate human error. Compared to software development for other autonomous vehicles, maritime software development, especially on aging but still functional fleets, is described as being in a very early and emerging phase. This introduces very large challenges and opportunities for researchers and engineers to develop maritime autonomous systems. Recent progress in sensor and communication technology has introduced the use of autonomous surface vehicles (ASVs) in applications such as coastline surveillance, oceanographic observation, multi-vehicle cooperation, and search and rescue missions. Advanced artificial intelligence technology, especially deep learning (DL) methods that conduct nonlinear mapping with self-learning representations, has brought the concept of full autonomy one step closer to reality. This paper surveys the existing work regarding the implementation of DL methods in ASV-related fields. First, the scope of this work is described after reviewing surveys on ASV developments and technologies, which draws attention to the research gap between DL and maritime operations. Then, DL-based navigation, guidance, control (NGC) systems and cooperative operations, are presented. Finally, this survey is completed by highlighting the current challenges and future research directions.

I. INTRODUCTION

This section introduces ASVs, explains how deep learning addresses maritime autonomy challenges, and defines the survey’s scope across ASV systems and operations.

  • ASVs support military, research, and commercial applications including surveillance, data collection, and communication hubs.
  • Deep learning methods address ASV problems through nonlinear system identification, model-free control, object detection, behavior prediction, and human-like decision making.
  • This survey summarizes and compares DL applications in ASVs, emphasizing NGC systems, cooperative operations, sensors, communications, challenges, and future directions.
  • Existing surveys largely cover ASV prototypes and NGC systems, while DL implementation for ASVs remains insufficiently discussed and comprehensively studied.

A. ASV Category

This section describes ASV categories, physical and onboard components, and the navigation, guidance, and control functions that support autonomous operation.

  • A. ASV Category: ASV types serve different missions, with naval vehicles prioritizing remote high-speed operation and payload capacity, while oceanographic vehicles prioritize navigation, energy conservation, and stable low-speed measurement.
  • A. ASV Category: Surface-vehicle motion can be represented in earth-fixed or body-fixed frames, with six rigid-body degrees of freedom including heave, sway, and surge displacement motions.
  • B. ASV Architecture: Sensors provide ASVs with self-state and environmental information, while communication links exchange information among onboard components, vehicles, control centers, and satellites.
  • 5) NGC System:: The NGC system integrates situation awareness, path planning, and vehicle actuation through navigation, guidance, and control subsystems.
  • 5) NGC System:: Navigation estimates vehicle state and surroundings, guidance generates obstacle-avoiding paths, and control produces the forces and moments required for motion.

IV. DEEP LEARNING MODELS AND TECHNIQUES

This section surveys supervised, unsupervised, and reinforcement learning methods and reviews neural architectures used for ASV perception and control tasks.

  • Supervised deep learning learns input-output mappings from labeled data and is applied to ASV navigation, guidance, and control.
  • MLPs approximate nonlinear functions and can estimate ASV controller uncertainties, while RBF, WNN, and FNN variants target convergence, localization, and uncertainty modeling.
  • 2) Convolutional Neural Network (CNN):: CNNs use convolution and pooling with sparse interaction and parameter sharing, supporting hierarchical feature learning for vision tasks.
  • 2) Convolutional Neural Network (CNN):: CNN-based classification assigns image labels, and established architectures such as AlexNet, VGG, GoogLeNet, Inception Net, and ResNet support feature extraction and recognition.
  • 2) Convolutional Neural Network (CNN):: Object detection either proposes regions before classification in two-stage methods or jointly predicts classes and bounding boxes in one-stage methods.
  • 2) Convolutional Neural Network (CNN):: Image segmentation classifies every pixel, with FCN and U-Net architectures producing dense predictions through convolutional features and upsampling or encoder-decoder paths.

3) Recurrent Neural Network (RNN):

The section introduces recurrent and related deep-learning methods for processing sequential, high-dimensional, noisy ASV data. It also describes autoencoders and GANs as tools for dimensionality reduction, denoising, and data generation.

  • 3) Recurrent Neural Network (RNN):: RNNs use historical information for future-behavior prediction but can suffer from vanishing and exploding gradients during training.LSTM adds memory and gates to address vanishing gradients, while GRU uses fewer gates and training parameters.
  • 1) AutoEncoder (AE):: Autoencoders learn compressed representations by reconstructing inputs while reducing noise and dimensionality.The navigation module applies AE to reduce noise and dimensions in original inputs.
  • 2) Generative Adversarial Networks (GAN):: GANs generate samples resembling training data through adversarial generator–discriminator training and can expand insufficient ASV navigation datasets.DC-GAN and WGAN variants are described as approaches for improving training stability or distribution measurement.
  • 1) Deep Reinforcement Learning (DRL):: Deep Q-networks extract features from large state and action spaces, while DDPG extends deep reinforcement learning to high-dimensional continuous actions.DQN is limited to discrete, low-dimensional action spaces; DDPG is model-free and off-policy.
  • 3) Data Processing:: Heterogeneous sensor streams differ in temporal and spatial resolution, data format, and geometric alignment, motivating compression, dimensionality reduction, and fusion.ASV sensing also faces noise and errors in position, orientation, and velocity measurements.

B. Navigation with Deep Learning Model

Deep learning navigation research addresses environmental perception from optical, radar, and remote-sensing data. The main challenges are limited or difficult data, complex scenes, variable ship geometry, and SAR-specific detection conditions.

  • B. Navigation with Deep Learning Model: Navigation perception methods use onboard, onshore, and satellite optical or radar data, including optical remote sensing and SAR images.The survey organizes methods by navigation data source and discusses optical and radar-image processing separately.
  • Environmental Perception: Optical imagery is constrained by time of day, cloud coverage, high-resolution processing demands, port clutter, and ships’ long, thin, arbitrarily rotated shapes.These challenges motivate work on complex backgrounds, multiscale ships, and small datasets.
  • Environmental Perception: Optical-image detection methods improve feature representation through multiscale feature extraction, feature fusion, data augmentation, and sea–land segmentation.Examples include bottom-up paths, skip-connection networks, and image-pyramid approaches.
  • Environmental Perception: Rotated bounding boxes reduce redundant regions for arbitrarily oriented ships compared with horizontal bounding boxes.A CNN-based approach independently predicts ship center, dimensions, and orientation.
  • Environmental Perception: Small ship-detection datasets can be addressed through pretraining on established open datasets or generating synthetic ship images with GANs.The survey also describes simulated SAR data and deeper networks as responses to limited ground truth.
  • Environmental Perception: SAR detection faces lower resolution, tiny clustered ships, arbitrary orientations, land-like scattering, and scarce extensively labeled datasets.These conditions have prompted research on multiscale detection and small-dataset detection.

2) State Estimation:

ASV state estimation and navigation data processing support guidance and control by extracting motion information from noisy, heterogeneous measurements. Deep learning is used for motion prediction, dimensionality reduction, and data fusion, while planning must respect task-specific constraints.

  • 2) State Estimation:: LSTM captures ship motion across frequency scales, and an attention mechanism is used to improve prediction accuracy.The described approach targets ship motion prediction.
  • 3) Data Processing:: Maritime navigation data are difficult to use because of noise from limited onboard loads, poor communication, and unexpected situations.DNN methods extract data representations before traditional multisensor processing to reduce dimensionality or improve quality.
  • 3) Data Processing:: An autoencoder reduces navigation-data dimensions during transmission, while CNN fusion extracts joint information from vessel motion, obstacles, and prior control behavior.The CNN produces a vessel operating-state vector called StateID.
  • 2) Local Path Planning:: Global path planning uses static maps or historical data, whereas local planning must adapt trajectories in real time to tasks and dynamic environments.AIS data create opportunities for global-path generation, while local planning addresses encounters among surface vehicles.
  • 3) Constraints:: ASV planning may need to account for geography, vehicle shape, kinematics, and dynamics depending on the planning scale and task.Dynamic constraints become important for precise small-scale tasks such as berthing.

B. Guidance with Deep Learning Model

Deep learning guidance methods extend ASV path planning beyond conventional collision-free search to energy-efficient routing, trajectory prediction, and multi-task planning. Local planning addresses dynamic encounters, COLREGS compliance, and the limitations of computationally demanding or oversimplified conventional methods.

  • B. Guidance with Deep Learning Model: DNN-based guidance has been applied to energy-efficient routing, future-path prediction, and optimal sequencing for multiple monitoring tasks.These tasks include Arctic ice-route optimization, multistep trajectory prediction, and a self-organizing-map approach to visit monitoring stations.
  • 2) Local Path Planning:: Conventional collision-avoidance methods can be limited by high computational complexity, oversimplified encounter assumptions, and control laws that do not adapt readily.These limitations are especially relevant in dynamic environments with multiple objects and COLREGS requirements.
  • 2) Local Path Planning:: DRL-based local planning learns responses through continuous interaction with uncertain environments using rewards for reaching destinations, avoiding collisions, and complying with COLREGS.Avoiding collisions can conflict with destination-oriented rewards when the ASV must deviate or reverse direction.
  • 2) Local Path Planning:: Multi-ship encounters require fixed-dimensional representations because the number of target ships changes continuously.Reported solutions use recent distance records, COLREGS-based regions, or LSTM and sequence conditional GAN models for encounter-mode decisions.

A. Definition and Key Problems

ASV modeling separates vessel motion into kinematics and dynamics, while controllers map reference errors and vehicle states into actuator commands. The surveyed framework covers standard control techniques and motion-control scenarios.

  • An ASV motion controller compares guidance references with actual outputs and sends commands to actuators such as thrusters and rudders.
  • ASV modeling distinguishes kinematics, which describes geometric motion, from dynamics, which analyzes the forces causing motion.
  • The vessel state combines inertial position and heading with body-fixed surge, sway, and yaw velocities, while transformation matrices relate the two frames.
  • The nonlinear vessel dynamics include mass and inertia, Coriolis and centripetal effects, drag, applied forces and torques, and environmental disturbances.
  • Thruster configuration maps the control vector to applied forces and moments through a control matrix, and the resulting state evolves according to a continuously differentiable update function.
  • Controllers may use classical, optimal, adaptive, intelligent, robust, or sliding-mode methods, either individually or in combination.

3) Actuation Capabilities:

ASV control spans stabilization, tracking, path following, maneuvering, and berthing, while practical systems must address uncertain models, physical constraints, disturbances, and limited sensing or communication. Deep-learning controllers are surveyed as model-free tools for estimating unknown dynamics or generating control signals across these scenarios.

  • Motion-control scenarios: Point stabilization fixes position and orientation, whereas target tracking follows static or dynamic targets without future target-motion information.
  • Motion-control scenarios: Trajectory tracking follows a specified path at a desired speed under time constraints, while path following imposes only a spatial constraint with constant forward velocity.
  • Motion-control scenarios: Maneuvering steers an ASV along a predefined path while speed may be treated separately, whereas berthing requires low-speed stopping near a berth.
  • Control problems: Model uncertainties arise from unknown dynamics, underactuation, high-speed maneuvering, sensor errors, and environmental disturbances.
  • Control problems: Control systems must handle input, output, velocity, performance, and communication constraints because violations can degrade tracking or cause failure and collisions.
  • Deep-learning control: DNNs estimate unknown parameters, terms, or functions, and are applied across point stabilization, target tracking, trajectory tracking, and berthing studies.
  • Deep-learning control: Berthing is especially difficult because low speed, winds, disturbances, and low signal-to-noise ratios make ship motion hard to represent with differential equations.

2) Control Signals Generation:

Data-driven DNN and DRL methods generate ASV control signals without requiring an a priori dynamic model. The surveyed approaches span actor–critic learning, direct policy generation, reward-based maneuvering, and transfer across berthing locations.

  • DNN and DRL map conditions to actions and can generate control signals without an a priori model.
  • An actor–critic DNN scheme learns an optimal policy and cost function simultaneously from ASV input–output data pairs.
  • For path following, DNN and DRL generate control signals without prior dynamic-model knowledge, while DRL learns policies through agent–environment interaction.
  • A DQN-based maneuvering controller combines CNN sensor fusion with rewards encoding destination, obstacle avoidance, target approach, speed, and attitude objectives.
  • A DNN berthing controller can adapt to different ports when their initial conditions resemble the original training data, using relative bearing and berth distance as key inputs.

VIII. DEEP LEARNING IN COOPERATIVE OPERATIONS

Deep learning supports cooperative ASV operations by estimating unknown dynamics and disturbances, generating control actions, and addressing constraints in leader-following and leaderless formations. The surveyed work also targets collision avoidance, communication reduction, connectivity, stability, and resource limits.

  • Cooperative control coordinates multiple ASVs through path-, trajectory-, or target-following controllers while maintaining formation geometry and completing shared tasks.
  • Leader-follower methods use a physical leader, whereas leaderless methods reach common values through local interaction and avoid single-point leader failure.
  • Leader-follower formation control: DL models in leader-follower formation control estimate unknown disturbances, dynamics, or velocities for path-guided and trajectory-guided coordination.
  • Leader-follower formation control: Trajectory-guided coordination addresses uncertainties alongside actuator saturation, stability, collision avoidance, connectivity, performance constraints, computational effort, and communication reduction.
  • Leaderless formation control: Leaderless path-guided control considers unknown dynamics and disturbances, input saturation, velocity estimation, communication reduction, cyberattacks, stability, and control-signal generation.
  • Leaderless formation control: DRL-based formation control uses rewards based on ASV velocities and formation-distance errors, enabling automatic and flexible formation adjustment.
  • Resource-aware coordination: Cooperative studies also reduce computational and communication demands through fewer learning parameters, event-triggered mechanisms, and distributed controller updates.
  • Resource-aware coordination: Safety-constrained coordination can map state safety constraints into control-input constraints and solve the resulting optimization with an RNN-based neurodynamic approach.

3) Data-driven IoT System:

This section surveys data-driven IoT and deep-learning applications for maritime monitoring, perception, state estimation, and energy-aware operation. It also identifies persistent challenges in maritime data availability, fusion, and harsh-environment perception.

  • Data-driven IoT System:: Onboard and onshore IoT can support autonomous-ship navigation through distributed decision-making and monitoring under a DL-type framework.
  • Data-driven IoT System:: Forecasting maritime traffic on spatial grids can support route design, congestion reduction, and more efficient traffic management.
  • Data-driven IoT System:: DRL-based SDN frameworks address the volume and dimensionality of heterogeneous data collected by cooperative ASV, UAV, and AUV ecosystems.
  • Data-driven IoT System:: LSTM models use AIS, GPS, fuel, rotation, temperature, and environmental data to estimate real-time fuel consumption, while reduced-space search minimizes voyage fuel use and total cost.
  • Data-driven IoT System:: Maritime environmental perception remains difficult because open, large-scale labeled marine-navigation datasets are unavailable and complex backgrounds challenge robust representations.
  • Data-driven IoT System:: DL improves detection and data fusion, but discontinuous or incomplete multisensor data and unresolved decision-making across navigation situations remain important challenges.

2) Guidance:

The guidance section covers DL for path planning, COLREG-compliant collision avoidance, decision-making, cooperative planning, and control. It emphasizes promise in uncertain maritime environments but also highlights simulation-transfer, reward-design, and hydrodynamic challenges.

  • Guidance:: For uncertain and incomplete environments, DNN-based heuristic planning can learn trajectory patterns beyond conventional planning with fully known constraints.
  • Guidance:: COLREG-compliant collision avoidance remains difficult because unknown disturbances and uncertain ship motions complicate risk assessment, while experiments often use simple encounters.
  • Guidance:: DRL reward functions strongly influence learned behavior but are commonly handcrafted, making a single reward function poorly suited to dynamic complex systems.
  • Guidance:: DRL can generate cooperative paths for multiple ASVs, but multi-vehicle operations require learning communication, cooperation, agent models, and formation behavior.
  • Guidance:: Hydrodynamic water-body interaction complicates ASV motion and control, and DRL controllers tested mainly in simulation often fail when training and testing distributions differ.
  • Guidance:: Complete real-world ASV control may combine DL with classical controllers such as MPC or PID to address hard system constraints.

2) Communications and Networks:

This section connects maritime networking, smart-port and urban-waterway applications, and the broader limitations of DL. The survey presents DL as relevant to maritime autonomy while emphasizing data, distribution-shift, reasoning, and reward-design constraints.

  • Communications and Networks:: The Internet of Ships links ASVs, UAVs, and other maritime objects, while remote operation still depends on reliable connectivity.
  • Communications and Networks:: DL-based perception can support automated ports through container identification, object recognition, situational awareness, monitoring, and surveillance.
  • Communications and Networks:: Urban waterways impose higher control and obstacle-avoidance requirements than open water, making DL and DRL potential approaches for crowded environments.
  • Communications and Networks:: Maritime autonomy research remains largely conceptual for MASS, and algorithms developed for small ASVs cannot be directly applied to large ships with larger sensing requirements.
  • Communications and Networks:: Supervised learning needs extensive labeled data, whereas DRL needs many experiments and handcrafted rewards, limiting learning with little external supervision.
  • Communications and Networks:: DL models cannot quickly adapt to changed data distributions with few examples, and deliberate multi-step problem solving remains difficult.
  • Communications and Networks:: The survey bridges DL and ASV research by reviewing DL models, NGC systems, cooperative operations, current challenges, and future research directions.
Loading 2210.08487v3…