PaperlayerView exact revision on arXiv

Source-linked AI summary

PPO vs. SAC: A Comparative Study for Cooperative Motion Control of Dual-Arm Space Robots

Nasim Fallahiarezoodar, Zheng H. Zhu

arXiv:2610.32034v1cs.AI

TL;DR

Dual-arm space manipulators could support debris removal and on-orbit servicing, but coordinating collision avoidance, orientation, and synchronized motion remains challenging. This paper compares PPO and SAC in a shared cooperative target-reaching framework. SAC outperforms PPO in the considered fixed-base simulations, though the study leaves observation-noise robustness and free-floating dynamics for future work.

  • Problem

    Existing work rarely combines target accuracy, collision avoidance, orientation, smoothness, synchronized completion, and unseen-target generalization in one dual-arm formulation.

    Source: Contributions · p. 3

  • Method

    The paper compares PPO and SAC in a coordinated dual-arm target-reaching framework using centralized state information and a multi-objective reward.

    Source: Reward Function Design · p. 8, §6 · p. 15

  • Results

    78.6% versus 7.4% mean success rate for SAC and PPO, respectively, with SAC also retaining most performance on unseen boundary targets while PPO largely collapses.

    Source: §6 · p. 16

  • Takeaways & Limitations

    Under the two-phase shaped reward, PPO can converge to a reward-loitering local optimum, so success rate is the primary task-performance metric.

    Source: Discussion · p. 15

    The evaluation is limited to fixed-base simulation with five seeds per algorithm; observation-noise robustness, free-floating dynamics, and sim-to-real transfer remain unevaluated.

    Source: Discussion · p. 15, §6 · p. 16

Abstract

from arXiv · show

The growing population of orbital debris has intensified interest in active debris removal and on-orbit servicing, for which dual-arm space manipulators offer improved grasping stability and load distribution relative to single-arm systems. However, coordinating two manipulators under inter-arm collision avoidance, orientation alignment, and sensing uncertainty remains challenging. This paper develops a coordinated reinforcement learning framework for cooperative motion planning of a dual-arm space manipulator in a target-reaching task with inter-arm collision-avoidance constraints. A centralized state representation combines joint-space, end-effector, and target-relative information, while a multi-objective reward function promotes target-reaching accuracy, orientation alignment, motion smoothness, synchronized task completion, and collision avoidance. Within this framework, Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) are compared as representative on-policy and off-policy algorithms, and the learned policies are further evaluated for generalization to previously unseen target configurations. Across five random seeds, SAC substantially outperforms PPO within the considered fixed-base simulation scenarios: it reaches an 80% success threshold in every seed (median 1.6M environment steps) whereas PPO does not reach it within a 4M-step budget, and it attains a higher mean success rate (78.6% versus 7.4%) with lower position and orientation errors and tighter bimanual synchronization. Both algorithms degrade on unseen boundary targets, but SAC retains most of its performance while PPO's collapses. The results also expose a failure mode of PPO: under the two-phase shaped reward, it tends to converge to a reward-loitering local optimum rather than committing to the precision task. This behavior highlights the sensitivity of on-policy learning to reward shaping.

1 INTRODUCTION

Dual-arm space robotics is motivated by debris-removal and servicing needs, but coordination under collision avoidance, redundancy, and sensing uncertainty remains difficult. The paper addresses gaps in integrated task coverage and algorithm comparison with a shared cooperative-learning framework.

Background and Motivation

  • Collision avoidance, kinematic redundancy, and smooth synchronized motion under sensing uncertainty make reliable dual-arm coordination central to practical viability.Dual-arm systems are described as improving force distribution, grasping irregular or tumbling objects, and reducing grasp failure relative to single-arm systems.Source: Background and Motivation · p. 2

Related Work

  • Classical controllers can depend on accurate models, while deep reinforcement learning offers interaction-based coordination; PPO and SAC represent on- and off-policy paradigms.SAC uses entropy regularization to promote exploration and stabilize learning in continuous action spaces.Source: Related Work · p. 2
  • Prior work rarely combines the full dual-arm objective set, and prior studies do not compare PPO with SAC under an identical formulation.The closely related single-arm study uses DDPG [18], while cited dual-arm studies use DDPG [19,20] or PPO alone [21].Source: Related Work · p. 2, Contributions · p. 3, Table 1 · p. 3

Contributions

  • The paper develops a shared cooperative-learning framework with centralized state inputs, multi-objective rewards, PPO–SAC comparison, and evaluation on unseen targets.The state combines joint-space, end-effector, and target-relative information; rewards address accuracy, orientation, smoothness, collision avoidance, and synchronized completion.Source: Contributions · pp. 3–4

2 PROPOSED SYSTEM

The proposed system is a planar, fixed-base dual-arm manipulator for cooperative target reaching, not a grasp-completion task. Its shared workspace has unequal orientation capability between the arms, especially near the outer boundary.

2 PROPOSED SYSTEM

  • Each four-joint arm has three rigid links and a two-finger gripper; success requires target reaching, not a completed grasp.The grasp bonus is an auxiliary shaping signal, and motion is planar with a 2D end-effector position and a single heading angle.Source: §2 · p. 4, Figure 1 · p. 4
  • The common workspace is the intersection of the individual arm workspaces, but right-arm orientation coverage is broader than the left arm’s, especially near the outer curve.The map is obtained by forward-kinematics sampling; Figure 2 identifies the bimanual reachability map.Source: §2 · p. 4, Figure 2 · p. 5
  • Boundary regions, particularly for the left arm, are more kinematically constrained, so targets are sampled within a conservative dexterous region.The spacecraft base is fixed, with free-floating dynamics left for future work.Source: §2 · p. 5

3 TECHNICAL PRELIMINARIES

The section formulates motion planning as an MDP and compares on-policy PPO with off-policy SAC. It explains PPO’s actor–critic learning and clipped policy updates.

3 TECHNICAL PRELIMINARIES

  • The MDP tuple defines the state, action, transition, reward, and discount components of the agent–environment interaction.The policy maps observed states to actions to maximize expected cumulative reward.Source: §3 · p. 5

Reinforcement Learning Algorithms

  • PPO updates from current-policy data, whereas SAC reuses replay-buffer experience; one representative algorithm from each paradigm is compared.The section characterizes on-policy methods as typically more stable and off-policy methods as generally more sample-efficient in high-dimensional continuous action spaces.Source: Reinforcement Learning Algorithms · p. 6

Proximal Policy Optimization

  • PPO uses an actor–critic architecture: its policy network outputs an action distribution, while its value network estimates expected return.PPO maximizes expected cumulative reward while seeking stable updates.Source: Proximal Policy Optimization · p. 6
  • PPO uses an advantage function to reduce gradient-estimate variance, and its critic approximates the state value from sampled trajectories.The advantage measures how much better an action is than the average value of its state.Source: Proximal Policy Optimization · p. 6
  • PPO constrains the policy probability ratio to [1 − ϵ, 1 + ϵ] in its clipped surrogate objective to prevent excessively large updates.The critic minimizes the TD value loss.Source: Proximal Policy Optimization · p. 6

Soft Actor-Critic

SAC adds policy entropy to expected return to encourage exploration in continuous action spaces. The study uses twin soft Q-networks and automatic temperature tuning.

Soft Actor-Critic

  • SAC augments expected return with policy entropy to encourage exploration in continuous action spaces.Its temperature parameter trades off reward against entropy.Source: Soft Actor-Critic · pp. 6–7
  • The study’s SAC variant trains twin soft Q-networks with slowly updated target networks, using the minimum of two critics to mitigate overestimation bias.Its policy is updated to maximize the entropy-regularized soft value.Source: Soft Actor-Critic · p. 7
  • Automatic temperature tuning targets a specified policy entropy, while replay-buffer reuse supplies SAC’s sample efficiency and can amplify rare successful transitions.The paper discusses this amplification in its results section.Source: Soft Actor-Critic · p. 7

4 PROBLEM FORMULATION

The problem is formulated as an agent–environment interaction for coordinated target reaching under task and safety constraints. Its centralized state combines arm, end-effector, and target-relative information, while actions increment joint angles.

Markov Decision Process Formulation

  • The motion-planning problem is instantiated through state, action, transition, and reward components to learn coordinated policies for both manipulators under task and safety constraints.Source: Markov Decision Process Formulation · p. 7

State and Action Spaces

  • The centralized state is a 24-dimensional vector organized into six groups, combining joint, end-effector, target-relative, and heading information.Source: State and Action Spaces · p. 7
  • The state includes the four absolute joint angles of each arm and the two-dimensional world-frame XY positions of both end-effectors.Source: State and Action Spaces · p. 7
  • Target-relative end-effector positions and end-effector velocities are included in the centralized state.Source: State and Action Spaces · p. 7
  • The state also includes each end-effector’s planar orientation and heading error, defined as normalized angular difference from its target orientation.Source: State and Action Spaces · p. 7
  • The eight-dimensional action space represents incremental joint-angle adjustments for the right and left arms, rather than absolute joint positions.The increments are specified in radians at each control step.Source: State and Action Spaces · p. 8

Transition Dynamics

The transition setup uses deterministic control steps, and the reward combines task, smoothness, proximity, collision, grasp, coordination, time, and success terms. Its staged task reward prioritizes position before orientation refinement.

Transition Dynamics

  • The experiments use deterministic dynamics without injected process or observation noise, hold each action for four simulation substeps, and limit episodes to 1000 control steps.Actions are scaled to at most 0.05 rad per joint per control step, and updated angles are clamped to joint limits.Source: Transition Dynamics · p. 8

Reward Function Design

  • The total reward sums task, smoothness, proximity, collision, grasp, asynchronous-progress, time, and success terms to promote accurate, aligned, smooth, safe, timely cooperative reaching.Source: Reward Function Design · p. 8
  • The two-phase task reward first drives position reaching with a small orientation hint, then disables position reward and strengthens orientation reward inside the success region.Each arm’s task contribution is disabled once its sub-task is complete, avoiding premature wrist rotation far from the target.Source: Reward Function Design · p. 8
  • The smoothness penalty combines end-effector velocity, joint velocity, and joint acceleration, with acceleration approximated by changes in joint velocity.The three terms are summed across both arms.Source: Reward Function Design · pp. 8–9
  • A proximity penalty discourages non-end-effector links from approaching targets, while a smooth bounded penalty applies when inter-link distance falls below the 0.02 m safety threshold.The collision penalty uses kc = 30.0 and a tanh formulation to provide a sharply increasing, differentiable signal without a hard-threshold instability.Source: Reward Function Design · p. 9
  • Each arm receives a +1.0 sparse grasp bonus on valid finger contact, while a −0.1 per-step asynchronous-progress penalty discourages neglecting the unfinished arm.Source: Reward Function Design · p. 9
  • A −0.005 per-step efficiency penalty and a progress-scaled −0.01 (t/Tmax) penalty encourage faster completion, with +300.0 awarded only at successful episode termination.The authors report that per-step completion bonuses induced reward-loitering near the success boundary rather than episode completion.Source: Reward Function Design · p. 9

Episode Termination and Success Criteria

Episodes end when both arms complete the task or when the 1,000-step horizon is reached. Completion requires both arms to satisfy position and heading tolerances simultaneously.

Episode Termination and Success Criteria

5 SIMULATION RESULTS

The evaluation compares PPO and SAC on cooperative target reaching with inter-arm collision avoidance, and assesses generalization to unseen target configurations. Claims are scoped to the fixed-base dual-arm simulation scenarios considered.

5 SIMULATION RESULTS

  • The evaluation compares PPO and SAC on target reaching with inter-arm collision-avoidance constraints.Source: §5 · p. 10
  • The study also examines whether learned policies generalize to previously unseen target configurations.Source: §5 · p. 10
  • Comparative claims apply to the fixed-base dual-arm simulation scenarios considered.Source: §5 · p. 10

Simulation Setup

Under a shared MDP and equal 4M-step budgets, SAC substantially outperformed PPO in convergence and in-distribution task performance. The comparison used five seeds, while algorithm hyperparameters were manually tuned.

Simulation Setup

  • Targets were resampled from the interior of the reachable region during training and from a shared fixed seed sequence during evaluation, with paired episode comparisons.All runs used the same MDP configuration, verified by a common configuration hash.Source: Simulation Setup · p. 10

Training Configuration

  • Both algorithms had 4M environment-step budgets across five seeds, but their hyperparameters were manually tuned rather than selected through a systematic equal-budget search.PPO used observation and reward normalization while SAC used raw observations.Source: Training Configuration · pp. 10–11, Table 2 · p. 11

Training Convergence

  • At a median of 1.6M steps, SAC reached the 80% success threshold in all five seeds; PPO reached it in none within 4M steps.The per-seed curves show this separation was consistent rather than driven by an outlier.Source: Training Convergence · p. 11, Figure 3 · p. 12

Task Performance

  • 78.6% versus 7.4% mean success: SAC outperformed PPO in-distribution, with lower position error, heading error, and per-episode collision rate.SAC completed both arms within a median of about 8 steps of each other, versus about 77 steps for PPO; this gap covers only episodes in which both arms latched.Source: Task Performance · pp. 11–12, Table 3 · p. 12

Generalization to Unseen Target Configurations

On unseen boundary targets, both algorithms degraded, but SAC retained substantially more success than PPO. Control experiments found that PPO’s main failure was at precision latching, and did not recover by changing its discount, extending training, or replacing the shaped reward.

Generalization to Unseen Target Configurations

PPO Failure-Mode Controls

  • PPO often reached within 5 cm but rarely completed the precision latch: both arms latched in 14.8% of episodes, versus 79.8% for SAC.Two-thirds of PPO episodes with one arm latched never completed the second arm, consistent with its large inter-arm completion gap.Source: PPO Failure-Mode Controls · p. 13
  • Neither matching SAC’s discount nor extending training recovered PPO: success was 4% at γ = 0.97, and longer runs plateaued at 15–22% by 16M steps or reached 1% at 50M.The controls indicate the reported PPO outcome was a converged plateau, not simply undertraining.Source: PPO Failure-Mode Controls · pp. 13–14
  • Removing per-step loiter income lowered success for both, but did not close the gap: SAC fell from 84% to 54%, while PPO fell from 10% to 0%.Source: PPO Failure-Mode Controls · p. 14
  • The controls point to a shaped-reward loitering optimum; PPO’s few-epoch on-policy updates discard rare successful batches, unlike SAC’s replay reuse.The single-seed controls are supplementary; the headline comparison remains the five-seed result.Source: PPO Failure-Mode Controls · pp. 14–15

Discussion

Within the considered fixed-base simulation scenarios, SAC outperformed PPO on task success, motion quality, and generalization to unseen boundary targets. The conclusion also identifies a reward-loitering failure mode for PPO and limits the study’s scope to simulation, with observation-noise robustness left unevaluated.

Discussion

  • The study is limited to a fixed-base simulation with five seeds per algorithm; observation-noise robustness was not evaluated, and free-floating dynamics, realistic contact, sim-to-real transfer, and hardware validation remain future work.The discussion also identifies noise robustness, free-floating-base extension, and sim-to-real transfer as future work.Source: Discussion · p. 15, §6 · p. 16

6 CONCLUSION

  • The paper combines centralized state information and a multi-objective reward for coordinated target-reaching, then compares PPO and SAC and evaluates policies on previously unseen target configurations.The reward promotes target accuracy, orientation alignment, smooth motion, synchronized completion, and inter-arm collision avoidance.Source: §6 · pp. 15–16
  • Across five random seeds, SAC reached an 80% success threshold in every seed (median 1.6M environment steps), while PPO did not reach it within a 4M-step budget; mean success was 78.6% versus 7.4%.SAC also produced more accurate and better-synchronized motions and retained most performance on unseen boundary targets, whereas PPO’s largely collapsed.Source: §6 · p. 16

Discussion

  • Under the two-phase shaped reward, PPO tended to converge to a reward-loitering local optimum that persisted across control variations; episode return was not monotone in task performance.The conclusion attributes this behavior to a limitation of on-policy sample reuse rather than a general deficiency of PPO.Source: Discussion · p. 15, §6 · p. 16

Related papers

Loading 2610.32034v1…