PaperlayerView exact revision on arXiv

Source-linked AI summary

A Unified Dynamics Framework for Reinforcement Learning and Classical Control of a Six-DOF Pipeline-Tracking ROV in NVIDIA Isaac Sim

Cheng Siong Chin, M. Venkateshkumar, Jianhua Zhang

arXiv:2610.04949v1cs.ROcs.AIcs.SYeess.SY

TL;DR

Underwater ROV policies are often trained and evaluated in one physics representation, leaving their behavior across deployment engines and against comparable classical controllers less established. This paper builds a shared-parameter pipeline-tracking architecture for vectorized training and Isaac Sim deployment, then compares learned and classical controllers. Under Coriolis-enabled dynamics, PPO, TRPO, and feedback linearization perform best overall; the evaluation also has a limited route and single-run scope.

  • Problem

    Prior pipeline-tracking work leaves open whether policies behave consistently across physics engines and outperform classical controllers under the same dynamics and guidance geometry.

    Source: §I · p. 2

  • Method

    A single USD scene supplies vehicle parameters to vectorized Fossen-dynamics training and an interactive Isaac Sim deployment, with learned and classical controllers evaluated through shared task geometry and scoring.

    Source: §I · p. 2

  • Results

    Under Coriolis-enabled dynamics, PPO, TRPO, and feedback linearization perform best overall in the shared 30-episode evaluation.

    Source: §XIII · p. 24

  • Takeaways & Limitations

    The controlled PPO and TRPO ablation supports retaining the Coriolis-centripetal term as the primary model for this task family.

    Source: §XIII · p. 24

    Experiments use one roughly 10 m L-shaped route with narrow rock randomization, and each controller comes from a single run.

    Source: B. Reinforcement learning versus… · p. 23

Abstract

from arXiv · show

Reinforcement learning controllers for underwater vehicles are usually trained against one physics representation and deployed against another, so reported performance does not always describe behavior outside training. This paper presents a pipeline-tracking architecture for a six-degree-of-freedom remotely operated vehicle (ROV) in which one Universal Scene Description (USD) scene supplies the real BlueROV2-Heavy mass, added-mass, damping, buoyancy, and thruster parameters to both halves of the system: a vectorized NumPy implementation of Fossen's marine-craft equations, and an interactive NVIDIA Isaac Sim deployment applying the identical equations as PhysX forces at every step. The Coriolis-centripetal term is the primary dynamics model in both branches; a controlled ablation on PPO and TRPO shows that including it does not destabilize either algorithm and modestly improves tracking, about 19 percent lower standoff RMS error for TRPO. Five reinforcement learning algorithms, PPO, soft actor-critic, TD3, DDPG, and TRPO, are trained against one environment, reward, and randomized evaluation harness through a checkpoint-compatibility layer scoring any policy with the same code. The pipeline is extended with six classical baselines, PID, sliding-mode, fuzzy logic, feedback linearization, model predictive control, and an adaptive neuro-fuzzy inference system, driven by the same guidance geometry and thruster allocation as the learned policies. Under Coriolis-enabled dynamics, PPO, TRPO, and feedback linearization reach the strongest combination of 100 percent success and competitive accuracy; PID, fuzzy control, and the neuro-fuzzy baseline also reach 100 percent success with looser tracking; DDPG and TD3 each show a specific, explainable failure mode rather than a general weakness of off-policy learning; and classical control remains a strong baseline against the best learned policies.

I. INTRODUCTION

Underwater ROV pipeline tracking combines six coupled degrees of freedom with nonlinear hydrodynamics, while prior work often leaves cross-engine transfer and comparisons across algorithms or classical controllers open. This paper addresses those gaps with a shared-parameter architecture spanning vectorized training and Isaac Sim deployment, alongside learned and classical controllers evaluated on common task geometry.

I. INTRODUCTION

  • Prior work rarely tests whether policies retain behavior across physics engines, across continuous-control algorithms, or against classical controllers using the same dynamics and guidance geometry.A shared architecture using the same Coriolis-coupled equations for training, deployment, and classical baselines had not been reported for pipeline tracking.Source: §I · pp. 1–2
  • The paper uses one USD scene and two independent code paths to supply shared vehicle parameters to vectorized training and interactive Isaac Sim deployment.The scene encodes mass, added mass, damping, buoyancy, and thruster allocation; the training branch uses a NumPy-vectorized Fossen model, while the deployment branch applies forces through PhysX.Source: §I · p. 2, A. The USD scene as the single… · p. 3, B. Training and deployment as two… · p. 4

A. Guidance and control for underwater vehicles

  • Line-of-sight guidance with PID or sliding-mode control is a long-standing AUV path-following approach, but new disturbances or actuator constraints typically require renewed stability analysis.These controllers do not adapt online to dynamics they were not designed against.Source: A. Guidance and control for underwater… · p. 2

B. Reinforcement learning for underwater tracking and following

  • Prior underwater RL work reports tracking, depth-holding, docking, and obstacle-aware methods, but commonly validates policies within their training dynamics and reports one algorithm per task.Classical baselines built from the same dynamics model are rarely reported alongside learned policies.Source: B. Reinforcement learning for… · p. 2

C. Off-policy and trust-region continuous control

  • The related algorithms use distinct update strategies: DDPG uses a deterministic actor-critic, TD3 adds twin critics and delayed updates, SAC optimizes entropy, and TRPO constrains policy changes.The paper motivates a comparison of these families using one environment, reward, and evaluation code.Source: C. Off-policy and trust-region… · p. 2

D. Simulation platforms and the sim-to-real gap

  • Isaac Sim’s USD scene graph can serve as a shared description of robot geometry and physical parameters, while MarineGym adds underwater hydrodynamics to its PhysX backbone.Isaac Gym and Isaac Sim address throughput through GPU-vectorized physics and observations; domain randomization varies physical parameters during training.Source: D. Simulation platforms and the… · pp. 2–3

E. The Omniverse, Isaac Sim, and Isaac Lab simulation stack

  • Omniverse provides underlying simulation infrastructure, Isaac Sim supplies the robotics environment, and Isaac Lab adds vectorized reinforcement-learning training tools.A generic workflow can train in Isaac Lab and deploy back to Isaac Sim, with an optional ROS 2 bridge to a physical vehicle.Source: E. The Omniverse, Isaac Sim, and Isaac… · p. 3, Figure 2 · p. 3

B. Training and deployment as two consumers of one model

IV. FOSSEN SIX-DEGREE-OF-FREEDOM VEHICLE MODEL

The vehicle model uses Fossen’s six-degree-of-freedom equations, with mass, Coriolis-centripetal coupling, damping, restoring forces, and thrust defining the dynamics. The Coriolis-centripetal term is retained as the primary model, and the model uses BlueROV2-Heavy parameters.

IV. FOSSEN SIX-DEGREE-OF-FREEDOM VEHICLE MODEL

  • The governing model includes the Coriolis-centripetal term as its primary dynamics representation; an ablation on PPO and TRPO found it stable and modestly beneficial to tracking accuracy.The same term is applied in Isaac Sim without double-counting the solver’s intrinsic rigid-body rotation handling.Source: §IV · p. 4

A. Mass matrix

  • The combined mass matrix adds diagonal added mass to a rigid-body mass matrix that includes centre-of-gravity offset coupling.Added-mass coefficients follow the reference script’s negative sign convention, with their positive magnitudes contributing to the combined mass matrix.Source: A. Mass matrix · p. 4

B. Coriolis-centripetal matrix

  • The Coriolis-centripetal matrix sums rigid-body and added-mass components, constructed from the corresponding mass-matrix blocks using a skew-symmetric cross-product form.With diagonal added mass, the resulting velocity terms include a roll-moment contribution from simultaneous sway and heave.Source: B. Coriolis-centripetal matrix · p. 4

C. Damping

  • Hydrodynamic damping combines linear and quadratic terms applied componentwise to body-frame velocity.The coefficients are specified in the cited damping table.Source: C. Damping · p. 5, Table II · p. 5

D. Restoring forces

  • The restoring-force vector uses net weight-minus-buoyancy and vehicle roll and pitch to determine its body-frame components.The parameter table reports weight, buoyancy, and net restoring force.Source: D. Restoring forces · pp. 5–6, Table III · p. 6

E. Thruster allocation

Eight normalized thruster commands are converted to forces and combined into a body-frame wrench, while vehicle kinematics and state integration advance the dynamics. The thruster layout and integration choices are specified for the training model and Isaac Sim deployment.

E. Thruster allocation

  • Each normalized command is clipped to [-1,1], scaled by the 40 N maximum force, and mapped through the scene’s 6×8 allocation matrix into a body-frame wrench.The eight-thruster layout uses four horizontal thrusters in a 45-degree X pattern and four vertical thrusters, one at each hull corner.Source: E. Thruster allocation · p. 6

F. Kinematics and time integration

  • Position and attitude rates use a body-to-world rotation matrix and an Euler-rate transform relating body-frame angular velocity to attitude change.The state variables include world-frame position and Euler-angle attitude.Source: F. Kinematics and time integration · p. 6
  • The Euler-rate transform is singular at pitch angles of ±90 degrees, so pitch is clamped to approximately ±85 degrees before evaluation.Source: F. Kinematics and time integration · p. 6
  • The standalone model integrates the full state with fixed-step fourth-order Runge-Kutta at 0.05 s, while Isaac Sim integrates the same forces through PhysX.The thruster allocation matrix is read from the USD scene.Source: F. Kinematics and time integration · pp. 6–7

V. REINFORCEMENT LEARNING ENVIRONMENT

The environment applies a shared Fossen-model setup, reward, and safety limits across five reinforcement-learning algorithms, with common guidance signals for six classical baselines. It defines the route task, observation and reward, episode conditions, and a projection gate that addresses corner-cutting progress.

V. REINFORCEMENT LEARNING ENVIRONMENT

  • The Fossen model runs in 64 independent NumPy vehicle instances, with the same environment, reward, and safety limits for all five learned algorithms and shared guidance signals for six classical baselines.Source: §V · p. 7

A. Episode length and control rate

  • 1,040 steps of 0.05 s give each episode a 52-second budget; policies act at 20 Hz, while Isaac Sim runs physics at 60 Hz and queries the controller every third step.Source: A. Episode length and control rate · p. 7

B. Observation

  • The 25-dimensional observation combines vehicle pose and velocity with route progress, cross-track and standoff information, heading alignment, and nearest-rock terms.Source: B. Observation · p. 7

C. Reward

  • The reward combines progress, tracking, heading, energy, and terminal terms, targeting 1.2 m overall standoff and 0.2 m horizontal standoff error.Source: C. Reward · p. 8

D. Termination and domain randomization

  • Episodes can fail through collision, seafloor contact, pipe-gap violation, excessive roll or pitch, or leaving operating bounds; rock positions and radii are jittered at reset.Success is recorded within 0.30 m of the route’s final waypoint; episodes can also end by timeout.Source: D. Termination and domain randomization · p. 8

E. A corner-cutting projection exploit

  • A projection gate blocks progress credit on the far side of the corner until the vehicle approaches the segment’s end vertex, rejecting a tested corner-cutting path while allowing genuine cornering.The exploit arose because projection onto the far segment could award progress without approaching the true corner vertex; the fix benefits learned and classical controllers.Source: E. A corner-cutting projection exploit · p. 8

VI. ON-POLICY OPTIMIZATION: PROXIMAL POLICY OPTIMIZATION

  • PPO is the on-policy baseline, and along with TRPO and feedback linearization it is among the strongest performers in the comparison with classical controllers.Source: §VI · p. 8

A. Tanh-squashed Gaussian policy

PPO uses a tanh-squashed Gaussian policy, generalized advantage estimation, and a clipped surrogate objective. Its training configuration includes a curriculum and a rolling-score checkpoint criterion reused for the other algorithms.

A. Tanh-squashed Gaussian policy

  • Tanh smoothly bounds sampled Gaussian actions, retaining nonzero policy gradients across the action range unlike hard clipping; all five algorithms use this action interface.Source: A. Tanh-squashed Gaussian policy · pp. 8–9

B. Generalized advantage estimation

  • Generalized advantage estimation uses temporal-difference residuals, rewards, value estimates, done flags, discounting, and trace decay to estimate advantages and value-function targets.Source: B. Generalized advantage estimation · p. 9

C. Clipped surrogate objective

  • PPO updates its policy with a clipped surrogate objective based on the importance-sampling ratio and estimated advantage.The total loss also weights value-function loss and policy entropy.Source: C. Clipped surrogate objective · p. 9

D. Training configuration and checkpoint selection

  • PPO and TRPO train on a short scene for the first 35 percent of the budget before switching to the full route; checkpoint selection guards against late entropy-driven drift using a rolling score.The highest-scoring checkpoint is saved, and the criterion is reused for SAC, TD3, and DDPG.Source: D. Training configuration and… · p. 9

VII. OFF-POLICY AND TRUST-REGION ALTERNATIVES

SAC, TD3, DDPG, and TRPO use the same environment and evaluation functions as PPO, with network capacity held constant. Replay-buffer training requires a curriculum adjustment, while the off-policy algorithms use distinct critic and policy-update designs.

VII. OFF-POLICY AND TRUST-REGION ALTERNATIVES

  • All four alternatives use PPO’s two-layer, 256-unit tanh network architecture and unchanged environment and evaluation functions through a compatibility layer.Source: §VII · p. 9

A. Design considerations for replay-buffer training

  • SAC, TD3, and DDPG disable the short-to-full-scene curriculum because replay buffers would mix transitions whose same observed progress can yield different rewards across phases.A 0.1 multiplier scales only the critic’s temporal-difference target for these algorithms.Source: A. Design considerations for… · p. 9

B. Soft actor-critic (SAC)

  • SAC adds an entropy bonus to return, uses twin critics and a soft Bellman target, and automatically tunes its temperature against target entropy.Source: B. Soft actor-critic (SAC) · p. 10

C. Twin-delayed DDPG (TD3)

  • TD3 combines target-policy smoothing, clipped double-Q learning, and delayed actor updates to address DDPG’s single-critic overestimation bias.The actor updates once per two critic updates.Source: C. Twin-delayed DDPG (TD3) · p. 10

D. Deep deterministic policy gradient (DDPG)

  • DDPG trains a deterministic actor from a single critic and bootstrapped target networks, with exploration noise added to the raw pre-tanh action.The target networks track online networks through Polyak averaging at rate 0.005.Source: D. Deep deterministic policy gradient… · p. 10

E. Trust region policy optimization (TRPO)

TRPO constrains policy updates through a KL-bounded natural-gradient step, while a shared compatibility layer supports consistent scoring across learned and classical controllers. The classical baselines share guidance and thruster allocation, with PID serving as the model-free reference.

E. Trust region policy optimization (TRPO)

  • TRPO approximates a KL-constrained natural-gradient update with conjugate gradient, scales the step to satisfy the constraint, and uses backtracking line search to accept it.The maximum KL is 0.01, with 10 conjugate-gradient iterations and Fisher damping of 0.1.Source: E. Trust region policy optimization… · pp. 10–11

F. Training budget and checkpoint-compatibility layer

  • A shared compatibility layer lets the same evaluation and checkpoint-selection code score learned policies and classical controllers through a deterministic act(observation) interface.The final training runs took 7.3 minutes for PPO, 7.4 for TRPO, 29.0 for DDPG, 34.4 for SAC, and 81.8 for TD3.Source: F. Training budget and… · p. 11

VIII. CLASSICAL AND MODEL-BASED CONTROL BASELINES

  • The six classical baselines use the same guidance signals, thruster allocation, and evaluation code as the five learned policies; ANFIS imitates another controller through supervised learning.Their per-channel gains and structural parameters are the stated differences among the classical baselines.Source: §VIII · p. 11, F. Training budget and… · p. 11

A. Proportional-Integral-Derivative (PID) Control

  • PID applies independently tuned per-channel gains to guidance errors, combines outputs into a desired 6-DOF wrench, and serves as the model-free reference controller.Integral action is clamped to limit windup during startup and large obstacle-induced cross-track errors.Source: A. Proportional-Integral-Derivative… · p. 11

B. Sliding-Mode Control (SMC)

The section describes sliding-mode and fuzzy controllers that map channel errors and error rates to thruster commands. Sliding-mode control smooths switching through a boundary layer, while fuzzy control uses a fixed rule base.

B. Sliding-Mode Control (SMC)

  • Sliding-mode control combines each channel’s error and error rate in a sliding surface, then uses tanh boundary-layer reaching instead of hard switching for smoother thruster commands.The smoothing trades finite-time convergence for a small, bounded steady-state tracking error.Source: B. Sliding-Mode Control (SMC) · pp. 11–12

C. Fuzzy Logic Control

  • The hand-designed fuzzy controller maps error and error rate through five triangular membership functions per input and a fixed 25-rule table to a weighted-average output.Its rule base is not adapted online.Source: C. Fuzzy Logic Control · pp. 12–13

D. Feedback Linearization (FL)

Feedback linearization cancels modeled nonlinear dynamics to produce decoupled linear error dynamics, while MPC searches a finite action horizon using the same Fossen model. Both approaches use model-based control, with MPC warm-started from feedback linearization.

D. Feedback Linearization (FL)

  • Feedback linearization cancels the modeled nonlinear terms at the current state, reducing the closed loop to linear, decoupled error dynamics that PD gains set from guidance errors.This simplifies tuning but depends on an accurate dynamics model, unlike purely error-driven PID and sliding-mode control.Source: D. Feedback Linearization (FL) · p. 13

E. Model Predictive Control (MPC)

  • MPC replans a 20-step action sequence with cross-entropy search, warm-started from a feedback-linearization trajectory, and applies the first action of the best sampled sequence.Its surge setpoint decreases as heading error from the look-ahead point grows, slowing the vehicle while turning through a route corner.Source: E. Model Predictive Control (MPC) · pp. 13–14

F. Adaptive Neuro-Fuzzy Inference System (ANFIS)

ANFIS is a learned per-channel Sugeno network trained to imitate PID rather than independently optimize pipeline tracking. Its results therefore characterize how closely a compact learned network reproduces PID behavior.

F. Adaptive Neuro-Fuzzy Inference System (ANFIS)

  • ANFIS uses learned Gaussian membership functions, rule-firing strengths, and linear consequents in a per-channel network structurally similar to the hand-designed fuzzy controller.Each channel takes error and error rate as inputs and forms nine rules from three membership functions for each input.Source: F. Adaptive Neuro-Fuzzy Inference… · p. 15
  • The network is trained by supervised gradient descent to imitate PID’s error-and-error-rate-to-output mapping, so it is a learned PID surrogate rather than an independently optimized control law.Its results indicate how closely the small network reproduces PID behavior, not whether it is an independently superior controller.Source: F. Adaptive Neuro-Fuzzy Inference… · p. 15
  • All controllers share guidance geometry, channel definitions, thruster allocation, and saturation handling, so reported performance differences reflect the control law rather than those shared components.ANFIS parameters are learned by minimizing output error against PID over 15 behavior-cloning episodes.Source: F. Adaptive Neuro-Fuzzy Inference… · p. 15

IX. DEPLOYMENT INSIDE NVIDIA ISAAC SIM

Isaac Sim deployment applies the vehicle’s hydrodynamic forces through PhysX, with timestep and added-mass reconciliation and a split Coriolis implementation. Evaluation uses the standalone Fossen model and shared scoring code.

IX. DEPLOYMENT INSIDE NVIDIA ISAAC SIM

  • Isaac Sim deployment computes body-frame Coriolis correction, damping, thrust, and restoring forces, sums them, and lets PhysX integrate the resulting motion.The controller derives body-frame velocities from the rigid body's world pose and velocities; PhysX gravity is disabled for this body.Source: §IX · p. 15

A. Timestep and added-mass reconciliation

  • ANFIS membership-function centres and widths are learned and may settle asymmetrically; yaw consequents span -2.479 to 2.162, and its error-rate width reaches 1.528.The passages contrast learned ANFIS parameters with the fuzzy controller’s symmetric, evenly spaced membership functions.Source: A. Timestep and added-mass… · p. 16, Table XVI · p. 16, Table XVII · p. 16
  • At 0.05 s (20 Hz), commands are held for three 60 Hz Isaac steps; added mass is folded into an effective mass matrix, while tracking-error metrics come from standalone Runge–Kutta.The standalone model remains the source of truth for tracking-error numbers.Source: A. Timestep and added-mass… · pp. 15–16
  • Deployment logs minimum standoff and maximum absolute roll or pitch, stopping on a detected flip or pipeline completion.Source: A. Timestep and added-mass… · p. 16

B. The Coriolis-centripetal term in the live controller

  • Training applies rigid-body and added-mass Coriolis terms, whereas Isaac Sim applies only the added-mass correction because PhysX already supplies the rigid-body term.This avoids double-counting the rigid-body effect; the ablation tests the full term in the training-side model.Source: B. The Coriolis-centripetal term in… · p. 16

X. EVALUATION PROTOCOL

  • Both evaluation phases use the standalone Fossen model, and the same evaluation code scores trained checkpoints and tuned classical controllers.The checkpoint-compatibility layer lets the evaluation functions score policies across algorithms.Source: §X · p. 16

A. Canonical episode and multi-episode statistics

The section defines the randomized evaluation and tracking measures, tests the Coriolis term, and compares learned and classical controllers, including distinct route-completion failures.

A. Canonical episode and multi-episode statistics

B. Tracking error and additional metrics

  • Tracking error is independently recomputed from logged positions as distance to the nearest point on the pipeline centerline, with RMSE, MAE, and maximum error reported.The logs also provide a squared normalized-thruster-command energy proxy and, for reinforcement-learning policies, steps and wall-clock time to first reach 90% deterministic success.Source: B. Tracking error and additional metrics · p. 17

A. Coriolis-term ablation (PPO and TRPO)

B. Reinforcement learning versus classical and model-based control

  • Under Coriolis-enabled dynamics, eight controllers complete the route; TD3 records 0% success despite 0.055 m RMSE because it parks at 5.00 m, while DDPG flips early.Tracking statistics are computed only over the route covered before TD3 parks, so the low RMSE does not indicate task completion.Source: B. Reinforcement learning versus… · p. 17, Table XIX · p. 18, Figure 6 · p. 19
  • The eight successful controllers track similarly on straight legs, but feedback linearization takes a wider arc at the corner while the other seven follow a tighter line.PID, SMC, fuzzy logic, MPC, and ANFIS remain within roughly one vehicle body-length of PPO and TRPO through the turn; depth stays in a narrow band.Source: B. Reinforcement learning versus… · p. 19, Figure 11 · p. 21, Figure 12 · p. 21
  • PPO converges to the 1.2 m standoff target fastest, within about 2.5 s; SMC and fuzzy logic converge more slowly, with RMSEs of 0.293 m and 0.243 m.All eight successful controllers stay above the 1.0 m hard floor; feedback linearization and MPC briefly undershoot the target to roughly 1.15 m.Source: B. Reinforcement learning versus… · pp. 17–18, Figure 9 · p. 20
  • PPO and feedback linearization peak at 2.0–2.2 m/s surge, whereas PID and ANFIS settle near 1.0 m/s; fuzzy control and SMC show sustained high-frequency roll- and pitch-rate activity.The paper distinguishes this attitude-rate activity from the SMC thruster-command chatter that its boundary-layer reaching law is designed to reduce; PID and ANFIS also share a sustained yaw-rate oscillation band.Source: B. Reinforcement learning versus… · pp. 20–21
  • PPO, feedback linearization, and MPC begin the final route leg around 4 s, while SMC and fuzzy logic begin around 10–14 s; their episodes last 19.4 s and 14.7 s.MPC has the largest early attitude transient, while SMC develops a roughly -40 degree mid-episode roll offset and fuzzy control shows persistent ±10–15 degree roll and pitch oscillations.Source: B. Reinforcement learning versus… · pp. 18–19, Figure 10 · p. 21

C. Other findings

The section identifies a guidance-geometry exploit that let controllers cut pipeline corners for progress reward, and reports a link between DDPG’s large actions and flips.

  • An unguarded projection let a diagonal path earn progress by cutting across the outside of a pipeline corner without reaching its vertex.A per-segment projection gate and look-ahead heading point close the gap for every controller.Source: C. Other findings · p. 22
  • The projection exploit came from shared guidance geometry, so its correction benefits every controller equally.The per-segment projection gate and look-ahead heading point address it.Source: C. Other findings · p. 22
  • DDPG’s action magnitude of 0.758 was roughly two to three times the 0.230 to 0.448 range of every other controller, consistent with its 30 percent flip rate.The paper links larger average thruster forces with sudden attitude changes that end in a flip.Source: C. Other findings · p. 22

XII. DISCUSSION

The discussion reports that feedback linearization, PPO, and TRPO combine 100 percent success with competitive tracking, while the Coriolis ablation supports including that term for the tested on-policy algorithms. It also identifies specific limits on the comparisons and concludes that classical and model-based control remain strong baselines.

A. What the Coriolis-term ablation shows

B. Reinforcement learning versus classical and model-based control

  • Feedback linearization, PPO, and TRPO combine a 100 percent success rate with RMSE values of 0.146 m, 0.142 m, and 0.231 m, respectively.PID, ANFIS, and fuzzy logic also reach 100 percent success, with RMSE values of 0.215 m, 0.211 m, and 0.243 m, respectively; MPC has 80 percent success and 0.177 m RMSE.Source: B. Reinforcement learning versus… · p. 23
  • DDPG shows a 30 percent flip rate and unusually large actions, while TD3 reaches a reproducible corner-parking local optimum but tracks its traversed route tightly.The TD3 failure is tied to this checkpoint’s exploration-noise and discount-factor setup, not evidence of a general off-policy weakness.Source: B. Reinforcement learning versus… · p. 23
  • Classical and model-based control remains highly competitive, and feedback linearization matches or exceeds PPO and TRPO tracking accuracy without training cost.The paper notes that feedback linearization has privileged access to the true dynamics model.Source: B. Reinforcement learning versus… · p. 23
  • The comparison is bounded by a single roughly 10 m L-shaped route, narrow rock randomization, single-run controllers, and a lightweight MPC planning budget.The paper also identifies independently tuned ANFIS, system-identified vehicle parameters, multi-seed evaluation, and physical validation as extensions or ways to strengthen the results.Source: B. Reinforcement learning versus… · pp. 23–24, §XIII · p. 25
  • The Coriolis-enabled comparison covers eight controllers’ flown trajectories over the full route and their body-frame linear and angular velocities in a canonical episode.The velocity plots use surge, sway, heave, roll rate, pitch rate, and yaw rate.Source: Figure 13 · p. 24, Figure 14 · p. 24

XIII. CONCLUSION

  • A single USD scene supplies Coriolis-coupled Fossen dynamics parameters to vectorized training and interactive Isaac Sim deployment.The conclusion reports the six classical and model-based baselines evaluated alongside PPO, TD3, DDPG, and TRPO under shared guidance geometry and thruster allocation.Source: §XIII · p. 24

ETHICS STATEMENT

  • The study involved underwater-vehicle simulation and no human or animal subjects, so ethics approval was not required.Source: ETHICS STATEMENT · p. 25

Related papers

Loading 2610.04949v1…