Sim-to-Real RL for ASVs using SysID

I. INTRODUCTION

• ASVs face nonlinear hydrodynamics, uncertain propulsion, and environmental disturbances; accurate modeling is difficult, while towing-tank or CFD identification creates a high entry barrier.Real-world SysID parameters may not match simplified simulator parameters, and heavy domain randomization can produce conservative policies and lower real-world performance.Source: §I · p. 1
• Station-keeping position RMSE fell 63.9% and heading MAE fell 40.1% versus analytical SysID on a physical BlueBoat under varying currents.Position RMSE changed from 1.074 m to 0.388 m; heading MAE changed from 17.30° to 10.37°.Source: §I · p. 1
• The contributions include an RL-focused ASV simulator and a real-to-sim pipeline using a CAD model and one open-water trajectory set instead of costly towing tanks or CFD.The work also evaluates simulator-trained path-following and station-keeping policies on a real ASV in a river environment.Source: §I · pp. 1–2

A. Marine Simulation

• The simulator targets surface vehicles and supports parallel RL environments, geometric wave displacement, and submerged-volume updates.The passage contrasts this with an existing ASV simulator limited to planar dynamics and constant-force current modeling.Source: A. Marine Simulation · p. 2

1) Domain Randomization:

• Domain randomization varies simulated environments to reduce overfitting, but heavy randomization can yield conservative policies; the passage recommends reserving it mainly for unmeasured dynamics and disturbances.The cited examples of unmeasured marine disturbances include wind and water.Source: 1) Domain Randomization · p. 2

2) Grounded Simulation Learning:

• Grounded simulation learning iteratively uses real-world data, deploys a policy, and refines simulator parameters; this work instead grounds dynamics without task-specific action transformation.The authors state that simulator hydrodynamics can be modified, allowing task-agnostic grounding without GAT.Source: 2) Grounded Simulation Learning · p. 2

III. METHODOLOGY

• The pipeline formulates efficient simulated dynamics and disturbances, fits simulator parameters using real-world SysID and trajectories, then defines MDP and domain-randomization formulations for zero-shot transfer.The overview describes CAD-derived initial hydrodynamic parameters refined with open-water trajectories using CMA-ES before RL training.Source: Figure 1 · p. 1, §III · p. 2

1) Hydrodynamics:

• The ASV is modeled natively as a 6-DOF PhysX system, while SysID and maneuvering use a 3-DOF relative-motion model.The model includes rigid-body and added-mass inertia, Coriolis-centripetal effects, damping, propulsion, and wave- and wind-induced forces.Source: 1) Hydrodynamics · p. 2
• The wave implementation targets semi-realistic, computationally efficient RL rather than high-fidelity CFD, using a finite-depth TMA spectrum based on JONSWAP.The spectrum incorporates depth attenuation and sets peak frequency from prescribed wind speed and fetch.Source: 1) Hydrodynamics · p. 2
• Geometry-based water-surface interactions estimate submerged hull volume, which determines buoyant force and restoring moments in heave, roll, and pitch.Hull draft is integrated across surface vertices; a piecewise-linear displacement coefficient converts draft to volume, clamped between zero and full-submergence volume.Source: 1) Hydrodynamics · pp. 2–3

2) Wind:

• The wind wrench uses relative wind speed and angle of attack with projected hull areas and surge, sway, and yaw coefficients fixed at nominal BlueBoat values.The stated values include air density 1.225 kg/m3, frontal and lateral areas 0.115 m2 and 0.163 m2, and coefficients 0.90, 0.20, and 0.30.Source: 2) Wind · p. 3, τ wind = · p. 3

3) Propeller Thrust:

• The propeller model linearizes thrust-coefficient behavior near nominal advance ratios to separate bollard pull from advance-inflow loss, represented by cu; crev captures reverse efficiency.The net active thrust is modeled as T ≈ T0(1 − cu ua).Source: 3) Propeller Thrust · p. 3
• Motor spool-up lag is modeled by a first-order low-pass filter on commanded thrust Tcmd with time constant τlag; during steady-state and coasting SysID, equivalent sequential commands eliminate motor lag.The real-world analytical baseline sets τlag nominally to 0.10 s.Source: 3) Propeller Thrust · p. 3, B. Real-World System Identification · p. 3
• Thruster forces account for forward advance-inflow loss, reverse-operation scaling, and port-starboard asymmetry, with lateral thruster position affecting inflow during yaw.A starboard motor imbalance factor is introduced to capture port-starboard propeller performance asymmetries.Source: 3) Propeller Thrust · p. 3
• The analytical baseline estimates damping and thruster coefficients from open-water BlueBoat trajectories, measures mass directly, and obtains yaw inertia and added-mass terms from CAD using BEM.The baseline exception is motor lag, nominally set to 0.10 s.Source: B. Real-World System Identification · pp. 3–4

1) 3-DOF Integration:

• The simplified 3-DOF model assumes minimal wave forces, port-starboard geometric symmetry, and negligible cross-coupling hydrodynamic forces at low speed.Source: 1) 3-DOF Integration · p. 4
• The model defines effective surge, sway, and yaw inertia terms by combining mass or yaw inertia with the corresponding added-mass terms.Source: 1) 3-DOF Integration · p. 4
• The expanded equations describe body-frame surge and sway acceleration and yaw-rate acceleration through damping, velocity coupling, and applied force or moment.The three equations are given for u, v, and r, respectively.Source: 1) 3-DOF Integration · p. 4

2) Parameter Extraction:

• Linear least squares uses forward and backward straight runs, a differential turn, and a circular single-thrust trajectory to estimate twin-screw ASV parameters.Coasting and steady-state surge isolate damping and forward/reverse thrust-efficiency parameters; turn and circular excitation isolate yaw and sway dynamics.Source: 2) Parameter Extraction · p. 4

C. Simulation System Identification

• CMA-ES refines the physical baseline by sequentially matching simulations to four recorded trials in 256 parallel environments, with parameter subsets locked as identified.Its search is bounded to ±30% of baseline estimates to retain physically plausible parameters and hydrodynamic damping ratios.Source: C. Simulation System Identification · p. 4
• The CMA-ES fitness minimizes the negative sum of bounded RBF similarity scores for state tracking, with weighted terms for path, velocity, heading, and yaw rate.Position and linear-velocity errors use σ = 1.0; heading and yaw-rate errors use σ = 0.75.Source: C. Simulation System Identification · p. 4
• The final parameter values after simulation SysID are summarized alongside real-world SysID estimates in Table V.The supplied passage identifies the comparison but does not provide the table values.Source: C. Simulation System Identification · p. 4

1) Action Space:

• The action is a continuous vector with one raw control value per propeller, mapped through asymmetric force limits and a deadband to target thrust commands.For the BlueBoat, the vector has two elements, one for each thruster; Table I is titled as the nominal action-space parameter values.Source: 1) Action Space · p. 4, Table I · p. 5

2) Observation Space:

• The actor’s shared vehicle state includes body-frame surge and sway velocities, yaw rate, accelerations, and a history of processed actions.The action history contains five previously processed action vectors sampled with a stride of two.Source: 2) Observation Space · p. 5
• Path-following actor observations include errors to the closest and two subsequent waypoints, cross-track error, and continuous sine/cosine encoding of path-relative heading error.Source: 2) Observation Space · p. 5
• Station-keeping observations encode target distance and body-frame position errors plus heading error as sine and cosine; the critic also receives privileged environmental information and noise-free actor states.Policy-observation terms are subject to uniform sensor noise; Table II is titled as the reward-function weights and hyperparameters.Source: 2) Observation Space · p. 5

• Both tasks share penalties on the first and second derivatives of thruster commands, while continuous task-specific objectives use Gaussian RBF kernels.The reward combines task objectives, action shaping, and termination conditions with weighting coefficients.Source: 3) Reward Objective Functions · pp. 5–6
• Path following rewards target forward velocity while minimizing cross-track and path-relative heading error, and terminates with a penalty beyond the cross-track limit.Continuous objectives use RBF terms for tracking errors and velocity along the path tangent.Source: 3) Reward Objective Functions · p. 5
• Station keeping minimizes positional and heading deviations from the starting state despite external disturbances, with a termination penalty when distance exceeds the allowed radius.The heading objective includes a deadband that allows minor deviations without penalty.Source: 3) Reward Objective Functions · pp. 5–6

• Training applies domain randomization to promote robustness to physical variances and environmental disturbances during policy transfer.Source: E. Domain Randomization · p. 6

1) Dynamics:

• Both tasks randomize initial surge, sway, and heading rate; station-keeping additionally randomizes position and heading offsets.The sampled velocity ranges are [−0.2, 0.2] m/s, [−0.1, 0.1] m/s, and [−3π/180, 3π/180] rad/s; station-keeping position offsets are within 0.5 m Manhattan distance and heading error up to 30π/180.Source: 1) Dynamics · p. 6
• At each episode’s start, hydrodynamic parameters and thrust-model parameters other than thrust limits are sampled uniformly.Source: 1) Dynamics · p. 6

2) Wind Disturbance:

• During training, initial wind direction is uniformly sampled over [−π, π], while steps 8,000–16,000 linearly scale initial wind-speed ranges from [0, 0] m/s to [0, 5] m/s for path-following and [0, 7] m/s for station-keeping.During each physics update, wind speed and direction evolve over time via an Ornstein-Uhlenbeck process.Source: 2) Wind Disturbance · p. 6

3) Wave Disturbance:

• The shared wave field uses γ = 3.3, 50 m depth, 4.0 m/s wind, 100 km fetch, and directional spreading factor 0.75.Wave generation is introduced via curriculum after 64,000 steps.Source: 3) Wave Disturbance · p. 6

4) Observation Noise:

• Gaussian noise perturbs actor observations but not processed action history or critic observations, which retain privileged state.Position noise has 0.025 m standard deviation; velocity and acceleration noise are 0.01 m/s and 0.02 m/s², and heading error and rate noise are 1°.Source: 4) Observation Noise · p. 6

F. Training Network

• PPO policies use asymmetric actor-critic MLPs and train in 256 parallel environments for 1,000 iterations.Both tasks share the stated training hyperparameters, while task-specific parameters are listed separately.Source: F. Training Network · p. 6

IV. EXPERIMENTS

• The four configurations are analytical SysID, proposed trajectory-matched CMA-ES grounding, single-zigzag optimization of nine parameters, and CMA-ES-scaled domain randomization.The Zigzag ablation uses real-world priors and ±30% bounds; CMA-ES DR scales variance up to ±30% according to optimized-parameter deviation from its prior.Source: §IV · pp. 6–7
• Proposed achieved the lowest average path-following cross-track RMSE, while station-keeping grounded policies outperformed Real-World and Zigzag notably outperformed CMA-ES DR.Source: Figure 2 · p. 7, Figure 3 · p. 7
• Field policies ran from matched starting poses in a river under calm weather, with three path-following runs and four station-keeping runs per policy.Different starting poses exposed policies to varying localized currents.Source: §IV · p. 7

V. RESULTS AND DISCUSSION

• 0.417 m cross-track RMSE for path following and 0.388 m position RMSE for station-keeping were achieved by Proposed, versus 0.423 m and 1.074 m respectively for Real-World.The station-keeping position RMSE reduction was 63.9%; Table VI summarizes real-world RMSE and MAE.Source: §V · pp. 7–8, Table VI · p. 8
• Single-trajectory Zigzag grounding became unstable after missing a turn and had the worst heading MAE among grounded methods.The authors attribute brittle performance in heading-change maneuvers to incomplete dynamic grounding.Source: §V · p. 8
• CMA-ES DR degraded tracking in both tasks relative to Proposed, with increased lateral drift in real-world currents.The authors report disproportionate variance on shifted sway and yaw damping parameters and negligible variance for cu.Source: §V · p. 8

VI. CONCLUSION

• The pipeline grounds simulator dynamics from four open-water trajectories and evaluates zero-shot deployment on a real BlueBoat in path-following and station-keeping tasks.Future work will examine individual hydrodynamic terms and their domain randomization for sim-to-real transfer.Source: §VI · p. 8