PaperlayerView exact revision on arXiv

Source-linked AI summary

SMaRT-Tug: Structured Multi-Agent Reinforcement Learning for Physics-Based Tugboat-Barge Collaborative Manipulation

Junkai Lu, Jiadong Zhao, Jiacheng Zhang, Wenqi Zhao, Hao Gen Chia, Qun Shen Png, Germaine Ee, Chengyang He, Yifeng Zhang, Nathanael Tan, Guillaume Sartoretti

arXiv:2609.07445v1cs.RO

TL;DR

Collaborative tugboat pushing is challenging because of coupled marine dynamics, disturbances, and contact-rich coordination, while scalable training environments and adaptable multi-tug approaches remain limited. The paper introduces a physics-based simulator and decentralized MAPPO with a structured control prior; its learned policy performs more reliably than PID and centralized PPO and generalizes to stronger waves and larger teams.

  • Problem

    Contact-based multi-tug pushing of shared barges is challenging under coupled marine dynamics and disturbances, while learning approaches face a shortage of scalable, physically realistic training environments.

    Source: §1 · p. 2

  • Method

    The paper combines a physics-based, GPU-accelerated simulator with decentralized MAPPO formulated as a residual controller over a position-holding structured control prior.

    Source: §1 · p. 2

  • Results

    The learned policy achieves more reliable maneuvering than PID and centralized PPO across straight-line transit, turning, and deceleration, and generalizes zero-shot to stronger waves and three- and four-tug teams.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The experiments support the potential of combining structured control priors with decentralized multi-agent learning for collaborative tugboat operations across increased disturbance and team size.

    Source: §1 · p. 2, §7 · p. 8

Abstract

from arXiv · show

Autonomous tugboating is central for automating maritime operations such as port logistics and vessel maneuvering, where multiple tugboats must cooperatively transport/manipulate a larger vessel. Collaborative pushing in this setting is challenging due to coupled hydrodynamics, low resistance, strong environmental disturbances, underactuated barge dynamics, and contact-rich interactions. Conventional control methods often rely on simplified models and fixed configurations, which limit their adaptability, while learning-based approaches are constrained by the lack of scalable and physically realistic training environments. We address these challenges by introducing a physics-based, GPU-accelerated simulation and learning framework for collaborative tugboat manipulation. Our simulator incorporates a customized buoyancy model, wave modeling, and hydrodynamic resistance, and supports large-scale multi-agent training under marine dynamics. In this simulator, we train a decentralized MAPPO (Multi-Agent PPO) policy augmented with a structured control prior (SCP) to improve training stability and maintain feasible pushing configurations. We evaluate our learned policy on straight-line transit, turning, and deceleration tasks, where we show that our decentralized framework yields more reliable and accurate maneuvering performance compared to a PID-based controller and a centralized PPO baseline. We further demonstrate zero-shot generalization to more challenging sea states and advanced maneuvers, as well as zero-shot scalability to larger teams of three and four tugboats despite training with only two agents.

1 Introduction

Collaborative barge pushing is a contact-rich, multi-tug manipulation problem made difficult by coupled marine dynamics and disturbances. The paper combines a physics-based, parallel simulator with decentralized MAPPO and a structured control prior, then evaluates the policy on maneuvering and generalization tasks.

Motivation and challenge

  • The study targets contact-based pushing because it enables direct tug–barge interaction with relatively light deployment requirements.The paper frames the task as collaborative pushing for autonomous barge maneuvering.Source: §1 · p. 2
  • Collaborative pushing couples multiple tugs’ contact interactions with the barge’s underactuated dynamics, inertia, low resistance, and wave disturbances.Existing controllers often use simplified models or fixed configurations, while learning-based multi-tug pushing and scalable, physically realistic training simulators remain limited.Source: §1 · p. 2

Contribution and results

  • The learned decentralized policy handles straight-line transit, turning, and deceleration more reliably than PID and centralized PPO, and generalizes zero-shot to stronger waves and teams of three or four tugs.The policy is trained with two agents; the reported larger-team tests use three and four.Source: §1 · p. 2

Contributions

  • The decentralized MAPPO policy uses a structured control prior (SCP) as a position-holding controller to stabilize training and maintain feasible tug–barge configurations.The policy is formulated as a residual controller over the SCP, allowing it to focus on interaction dynamics rather than learning the entire behavior from scratch.Source: §1 · p. 2
  • The simulator supports parallel reinforcement-learning training with voxelized buoyancy, wave modeling, and customized hydrodynamic resistance.It is developed on NVIDIA’s IsaacLab.Source: §1 · p. 2

2 Related Work

Prior marine control, learning-based maritime manipulation, and physics-rich simulators each address parts of collaborative tugboat–barge pushing, but the related work describes gaps in adaptable multi-tug control and scalable training environments.

2.1 Marine Vessel Control and Tug-assisted Maneuvering

  • Traditional marine controllers are interpretable and suited to single-ship control, but often depend on simplified models, specified vessels, fixed tug configurations, or limited team sizes.Model-based methods may require hydrodynamic coefficients, contact geometry, environmental forces, and thrust limits, and may need redesign as arrangements or disturbances change.Source: §2.1 · pp. 2–3

2.2 Learning-based Control for Marine Cooperative Manipulation

  • Learning has been applied to marine control and related berthing, docking, tracking, and tugboat tasks, while learning-based control can complement analytical models under complex hydrodynamics.Direct research on contact-rich manipulation of shared objects remains limited; formation control and AIS-based imitation do not cover the multi-agent contact interaction distribution required for multi-tug coordination.Source: §2.2 · p. 3

2.3 Simulators for Multi-robot Learning in Physics-rich Environments

  • General-purpose physics and scene-building platforms, and maritime-specific simulators, do not provide the same combination of marine modeling and open-ended, massively parallel cooperative tugboat–barge RL training.The related work notes that particle-based fluid simulation may add computational overhead in large-scale training, while maritime tools often target more constrained evaluation or simulation environments.Source: §2.3 · p. 3

3 Problem Formulation

The paper formulates tugboat–barge maneuvering as decentralized cooperative manipulation: agents use local observations and jointly control a shared underactuated barge. Straight-line transit, turning, and deceleration are instances of tracking commanded planar motion.

System Setup

  • Two tugboats physically interact with one barge to execute commanded planar maneuvers, with the barge as the manipulated object and tugboats as controlled agents.The setup combines persistent contact, coupled marine dynamics, and coordination through a shared underactuated object.Source: §3 · p. 3
  • Straight-line transit, turning, and deceleration are treated as task instances of tracking a commanded planar barge motion, rather than unrelated control tasks.The commanded motion may vary over time.Source: §3 · p. 4

Observation, Action, and Reward

  • Each tugboat acts from ego-centric local observations under decentralized execution, while training uses a self-first global critic state with privileged physics parameters.Agents share policy parameters; each outputs forward-velocity and yaw-rate commands, with lateral correction supplied by the SCP.Source: §3 · p. 4

4 Physics-Based Simulation Framework

The simulator combines marine-specific physical models with GPU-accelerated parallel simulation for cooperative tugboat manipulation. Its wave and buoyancy models support batched computation, while explicit hydrodynamic resistance is applied to the barge.

4 Physics-Based Simulation Framework

  • The simulator integrates GPU-accelerated simulation with voxel-based buoyancy, realistic waves, and calibrated hydrodynamic resistance to capture vessel interaction while supporting large-scale parallel training.Vessels are rigid bodies with contact interactions resolved by the physics solver.Source: §4 · p. 4
  • The simplified Gerstner-wave model represents water as an explicit height field, enabling direct, batched queries for buoyancy computation.It neglects horizontal particle displacement; wave components specify amplitude, direction, frequency, and phase.Source: §4 · p. 4
  • Submerged voxels approximate displaced volume, with buoyancy applied at the corresponding center of buoyancy.Voxel resolution can be adjusted to trade computational cost against accuracy.Source: §4 · p. 4
  • Explicit hydrodynamic resistance models low-speed barge maneuvering, while tugboats use simulator-side filtering of commanded velocities instead.The barge hull model combines linear hydrodynamic derivatives and cross-flow drag terms.Source: §4 · p. 4

5 Structured Multi-Agent Learning Framework

The learning framework combines a structured control prior with a residual multi-agent policy, using local observations for decentralized actors and centralized state during training. Joint two-tug training uses parallel simulation across varied maneuvers, initial conditions, and selected task and physical variations.

5 Structured Multi-Agent Learning Framework

  • The structured control prior regulates each tug’s relative position and alignment to the barge, keeping it near designated contact regions without solving the full maneuver task.It provides nominal tug-local velocity references and lateral correction for cooperative contact and alignment during pushing, turning, and stopping.Source: §5 · p. 5
  • Each tug’s residual policy uses an ego-centric local observation and primarily modulates forward speed and yaw, while the prior supplies lateral correction.The executed control combines the nominal command with the learned residual correction.Source: §5 · p. 5
  • Shared-parameter MAPPO trains decentralized actors on local observations and role indicators, with a privileged centralized critic used only during training.The critic inputs are arranged self-first to retain agent-specific perspectives while using global coordination information.Source: §5 · p. 5
  • Joint two-tug policies train in 128 parallel environments across varied maneuvers, initial conditions, and selected task-level and physical variations.Continued training and moderate task-difficulty curricula can support specialization or convergence; these staged procedures improve optimization stability and sample efficiency.Source: §5 · p. 5

6 Experiments

MAPPO outperforms the baselines across basic maneuvering tasks, and the full maneuver combines those skills in stable sequential execution. The results also show SCP’s importance for convergence and report generalization to varying wave conditions and larger tug teams.

6.1 Experimental Setup

  • Experiments cover straight-line transit, turning, deceleration, and a combined waypoint maneuver, comparing MAPPO with a proportional-only PID controller and centralized PPO.The basic tasks assess velocity tracking and recovery, turning displacement and recovery, and residual velocity after deceleration.Source: §6.1 · p. 5

6.2 Results

  • MAPPO supports initial misalignment up to 80° versus 5° for the PID baseline, achieves the lowest error in large-angle turning, and has the lowest residual velocities in both deceleration settings.For smaller-angle turning, MAPPO substantially reduces displacement compared with PPO.Source: §6.2 · p. 6
  • MAPPO achieves the strongest overall accuracy and adaptability across the basic tasks, outperforming the PID-based controller and centralized PPO baseline.The comparison covers straight-line transit, turning, and deceleration.Source: §6.2 · p. 6, Table 1 · p. 6
  • The full maneuver integrates transit, turning, deceleration, and repositioning with stable, continuous stage transitions, including pushing from an unseen barge side.The sequence uses a high-level controller that switches between stage-specific policies without a smoothing mechanism.Source: §6.1 · p. 5, §6.2 · p. 6, Figure 3 · p. 7
  • Nominal SCP yields stable, accurate control; half-strength SCP leaves lateral drift, while zero and decaying SCP fail to converge to a usable policy.The ablation varies the x- and yaw-axis SCP gains while keeping the y-axis gain active.Source: §6.2 · pp. 6–7, Figure 4 · p. 7

6.3 Generalizability and Scalability

  • Training with 0.8 m waves performs well from 0.0–0.8 m and remains more robust at 1.2 m than training in calm conditions, though all policies struggle at 1.2 m.The wave-amplitude comparison evaluates velocity-tracking MSE without retraining.Source: §6.3 · p. 7, Table 2 · p. 8
  • Without additional training, the two-tug policy reduces turning displacement from 111.09 m to 21.35 m with three tugs, and four tugs complete the maneuver with closer waypoint tracking.The four-tug maneuver includes in-place turning without repositioning, as shown in Figure 5.Source: §6.3 · p. 7, Figure 5 · p. 8

7 Conclusion

The paper presents a physics-based simulator and MAPPO with a structured control prior for collaborative tugboat manipulation under coupled marine hydrodynamics. Its experiments report more reliable maneuvering than the baselines, sequential composition of controllers, and zero-shot generalization to unseen configurations and larger teams.

7 Conclusion

  • The framework combines a physics-based surface-vessel simulator with MAPPO and a structured control prior for position-holding to support stable contact-based manipulation under coupled marine hydrodynamics.The conclusion presents the approach as a learning-based framework for collaborative tugboat manipulation of a barge.Source: §7 · p. 8
  • Experiments show more reliable maneuvering than the PID-based controller and centralized PPO baseline, composition across sequential command phases, and zero-shot generalization to unseen configurations and larger teams.The paper identifies straight-line transit, turning, and deceleration as basic evaluation tasks.Source: §7 · p. 8
  • The results suggest structured multi-agent reinforcement learning is a promising direction for autonomous tugboating and collaborative marine manipulation.Source: §7 · p. 8

A Observation, Action, and Reward

The appendix specifies the agents’ observations, actions, and task rewards, alongside a simplified wave model designed for efficient height queries. The wave approximation removes horizontal particle displacement, making the surface explicit while retaining most water-height variation.

A Observation, Action, and Reward

  • The appendix specifies reward terms for straight-line transit, turning, and deceleration, including orientation, formation, and drift errors for maneuvering.For deceleration, it also defines a speed reference, a brake reference that decreases over time, an exponentially smoothed deceleration estimate, and a stop-hold condition.Source: A Observation, Action, and Reward · pp. 10–11
  • The actor observation structure is shared across straight-line transit, turning, and deceleration; the appendix also lists the centralized critic state and normalized residual action space.For H=18, the history features contribute to a total observation size of 575.Source: Table 3 · p. 10, Table 4 · p. 10, Table 5 · p. 10

B.1 Details of Wave Model

  • The classical Gerstner surface is implicit because horizontal position depends on wave displacement, so fixed-location height queries require solving a nonlinear equation.The simplified model removes horizontal displacement to produce an explicit height field; the text says this loses only minor surface-geometry details, such as crest sharpness.Source: §B.1 · p. 11
  • The simplified wave field uses sampled propagation directions and frequency components, with spectral amplitudes derived from a TMA spectrum to represent wind-driven sea states and water-depth effects.The passage identifies the TMA spectrum as a finite-depth extension of JONSWAP.Source: §B.1 · p. 11

B.2 Details of the Buoyancy Calculation

Buoyancy is computed from a voxelized approximation of each vessel by identifying submerged voxels and summing their buoyant forces. The resulting model is parallelizable on the GPU, with voxel resolution trading computational cost against accuracy.

B.2 Details of the Buoyancy Calculation

  • Each vessel is discretized into volumetric cuboids, and submerged voxels estimate displaced water volume for the buoyancy calculation.Figure 7 depicts submerged voxels in blue and unsubmerged voxels in green.Source: §B.2 · p. 11
  • A voxel is marked submerged by comparing its center height with the local water-surface height.The water height is evaluated at the voxel center’s horizontal position and the current time.Source: §B.2 · p. 11
  • Each submerged voxel contributes an upward buoyant force under Archimedes’ principle; the total is summed across voxels and applied at the center of buoyancy to generate torque about the vessel center of mass.The center of buoyancy is defined as the volume-weighted centroid of the submerged voxels.Source: §B.2 · p. 11
  • The voxel-based calculation is fully parallelizable for GPU simulation, and voxel resolution trades computational cost against buoyancy accuracy.Source: §B.2 · p. 11

B.3 Details of the Hydrodynamic Resistance Model

The barge resistance model combines a unified hull-force formulation with empirical resistance terms for low-speed maneuvering. A correction factor calibrates simulator resistance against a scaled water-tunnel reference, which the authors treat as trend-level calibration rather than full-scale validation.

B.3 Details of the Hydrodynamic Resistance Model

  • The barge hull model represents surge, sway, and yaw using linear hydrodynamic derivatives and cross-flow drag terms.The formulation retains conventional linear maneuvering derivatives while augmenting sway and yaw with cross-flow drag contributions.Source: §B.3 · pp. 11–12
  • The resistance estimate uses conventional resistance decomposition, including friction and roughness allowances, a stern-shape-based form factor, and an empirical air-resistance allowance.The friction coefficient follows the ITTC-1957 friction line, and the roughness allowance follows the ITTC-1978 method.Source: §B.3 · p. 12
  • The model targets low-speed barge maneuvering, treats wave-making effects as secondary, and does not separately parameterize appendage resistance.The authors describe it as a practical low-speed component, not a fully predictive solver; the water-tunnel result is a trend-level calibration reference rather than strict full-scale validation.Source: §B.3 · p. 12
  • After a global correction factor of 2.5, simulator resistance closely agrees with the scaled water-tunnel reference over the low-speed maneuvering regime.Figure 8 compares the water-tunnel reference with simulator resistance after the correction.Source: §B.3 · p. 12

C Detailed Experimental Settings

The appendix identifies the experimental settings and controller parameters for the basic tasks. It also lists the SCP gain variants used in an ablation and the gain-decay settings used during training and inference.

C Detailed Experimental Settings

Loading 2609.07445v1…