PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Deep Reinforcement Learning for Robotic Motion Planning via Task Decomposition

Yuanzhi He, Victor Romero-Cano, Jiayu Xu, Juan David Hernández, Gualtiero Colombo

arXiv:2610.35494v1cs.RO

TL;DR

Obstacle-aware manipulation enlarges observation and action spaces, making scratch training sample-intensive and often slow or unstable. The framework combines transfer from simpler tasks using only first-layer critic parameters with staged rewards for multi-step objectives. Transfer learning improves convergence speed and final performance on Push and Slide, while reward redesign accelerates early-stage Pick-and-Place learning; combining TL and RR still yields substantial training acceleration.

  • Problem

    Obstacle-aware manipulation enlarges observation and action spaces, making scratch training sample-intensive and often slow or unstable.

    Source: §I · p. 1

  • Method

    The framework combines transfer from simpler tasks using only first-layer critic parameters with staged rewards for multi-step objectives.

    Source: §I · p. 1

  • Results

    Transfer learning improves convergence speed and final performance on Push and Slide, while reward redesign accelerates early-stage Pick-and-Place learning; combining TL and RR still yields substantial training acceleration.

    Source: §V · p. 7

  • Takeaways & Limitations

    The framework is validated on a real Franka Emika FR3 robot as a qualitative sim-to-real feasibility demonstration.

    Source: §V · p. 8

    For obstacle Pick-and-Place, geometric mismatch means transfer learning alone does not provide clear improvements.

    Source: §V · p. 7

Abstract

from arXiv · show

Training robotic manipulation policies in simulation is widely adopted due to safety and cost efficiency; however, tasks involving obstacles remain under-performing and often converge slowly under Deep Reinforcement Learning (DRL). To address this challenge, we propose a task decomposition framework for DRL that integrates Transfer Learning (TL) with Reward Redesign (RR) to improve learning efficiency and robustness in obstacle environments. The TL strategy selectively transfers only the first-layer critic parameters across structurally related tasks, since that layer captures general state--action representations while deeper layers encode reward- and obstacle-specific value geometry; RR decomposes tasks into staged sub-objectives to provide denser optimization signals. Experiments on the PyBullet Panda platform show consistent gains in obstacle settings. On Push with obstacle, our method achieves $0.83 \pm 0.05$, outperforming the strongest baseline (Truncated Quantile Critics (TQC), $0.77 \pm 0.06$) by approximately $7.8\%$. On Slide with obstacle, performance reaches $0.95 \pm 0.02$, exceeding Soft Actor-Critic (SAC) ($0.84 \pm 0.06$) by about $11\%$ success rates. In addition to higher final performance, forward transfer reduces convergence time by up to $70\%$ in obstacle tasks, and RR shortens sparse-reward Pick-and-Place training from roughly $0.75$M to $0.20$M steps ($\approx73\%$ reduction). The best policy is deployed on a real Franka Emika FR3 robot as a qualitative feasibility demonstration.

I. INTRODUCTION

Obstacle-aware manipulation increases the demands of DRL training, while scratch training can require many samples and converge slowly or unstably. The paper combines transfer learning with staged reward redesign and evaluates the framework across manipulation tasks and settings.

I. INTRODUCTION

  • Obstacle-aware goal-conditioned policies must reach goals, interact with objects, and avoid collision regions without an explicit geometric planner.The larger observation and action spaces make scratch training sample-intensive and often slow or unstable.Source: §I · p. 1
  • Few-shot transfer with reward redesign improves success rates and reduces training time across most task settings, with obstacle Pick-and-Place as an exception.The evaluations cover Push, Pick-and-Place, and Slide in obstacle-free and obstacle environments.Source: §I · p. 1
  • The framework combines few-shot transfer learning and reward redesign to improve learning efficiency and performance for obstacle-aware manipulation.Its staged rewards decompose multi-step objectives into interpretable sub-task rewards.Source: §I · pp. 1–2
  • The transfer strategy reuses only first-layer critic parameters, and the paper compares first-layer, all-layer, and scratch training.The approach also examines when transfer helps and when it does not.Source: §I · pp. 1–2
  • The framework is evaluated through systematic transfer studies and a real-world deployment on a Franka Emika FR3 robot.The introduction describes the deployment as demonstrating real-world feasibility.Source: §I · p. 2

II. RELATED WORKS

Prior work applies DRL and transfer learning to robotic manipulation, but transfer under geometric constraints and increasing task complexity has received less attention. The paper situates its approach among off-policy algorithms, sparse-reward methods, and perception-focused foundation models.

II. RELATED WORKS

  • Prior manipulation transfer studies reuse parameters and low-level representations, but less attention has been given to transfer under geometric constraints or increasing task complexity.Selective sharing has been used in few-shot and zero-shot transfer strategies [15].Source: §II · p. 2
  • Off-policy methods such as DDPG, TD3, SAC, and TQC leverage replay to improve sample efficiency, while PPO [18] is described as robust but less sample-efficient.HER replaces the intended goal with the achieved goal in replay to improve sparse-reward learning; TQC is the paper's backbone and strongest experimental baseline.Source: §II · p. 2
  • Foundation models such as CLIP and DINO are used in robotics mainly for perception and high-level representations rather than low-level control.The passage notes domain-shift and computational-overhead concerns for real-world deployment.Source: §II · p. 2
  • The paper targets the interaction-data and training-time demands of scratch training for complex, obstacle-rich tasks through structured knowledge reuse.Its related-work framing emphasizes transfer-based sample efficiency.Source: §II · p. 2

A. Problem Formulation

The paper models each manipulation task as a goal-conditioned Markov Decision Process and defines learning as maximizing expected cumulative return under a goal-conditioned policy.

A. Problem Formulation

  • Each task is a goal-conditioned Markov Decision Process for a 7-DoF Franka Panda arm in obstacle-free or obstacle environments.The formulation builds on a goal-conditioned robotics benchmark.Source: A. Problem Formulation · p. 2
  • The MDP comprises continuous state and action spaces, a goal space, transition dynamics, goal-conditioned rewards, and a discount factor.The reward is conditioned on goal g, and γ lies in (0, 1].Source: A. Problem Formulation · p. 2
  • The objective is to learn a goal-conditioned policy that maximizes expected cumulative return.The policy is written as πϕ(at | st, g).Source: A. Problem Formulation · p. 2

B. Augmenting TQC with Transfer Learning

The framework transfers selected critic parameters between related manipulation tasks, reusing first-layer representations while leaving deeper task-specific value layers to be learned for the target.

B. Augmenting TQC with Transfer Learning

  • Tasks in the transfer set share state and action representations but differ in goal definitions and reward structures.Source: B. Augmenting TQC with Transfer Learning · p. 2
  • Fig. 1 shows Reach, Push, Pick&Place, and Slide in PyBullet Panda simulation, each with one red-cube obstacle.Source: Figure 1 · p. 3
  • Fig. 2 depicts first-layer critic weights and biases transferred between source and target tasks in forward and reverse directions across obstacle-free and obstacle environments.Source: Figure 2 · p. 3
  • Only the critic’s first layer is transferred because it captures general state–action representations, whereas deeper layers reflect reward geometry and obstacle layout.Higher layers are randomly initialized and optimized normally, without freezing or a special fine-tuning schedule.Source: B. Augmenting TQC with Transfer Learning · pp. 2–3
  • Source tasks are chosen by a simple-to-complex ordering among tasks with compatible state/action spaces and contact dynamics, favoring similar object-interaction patterns.Source: B. Augmenting TQC with Transfer Learning · p. 3
  • The evaluated transfer settings include single- and multi-step forward transfer, reverse transfer, and neutral transfer when reuse does not improve over training from scratch.Multi-step forward transfer passes parameters sequentially across tasks of increasing structural complexity.Source: B. Augmenting TQC with Transfer Learning · pp. 3–4

C. Decomposing Multi-step Tasks with Staged Reward Formulation

The staged reward formulation decomposes multi-step objectives into intermediate sub-goals, providing denser feedback during early training and guiding policies through task stages.

C. Decomposing Multi-step Tasks with Staged Reward Formulation

  • Staged reward redesign decomposes a multi-step objective into intermediate sub-goals to provide denser, more informative early-training signals and stabilize convergence.Source: C. Decomposing Multi-step Tasks with… · p. 4
  • The redesigned reward is a weighted sum of sub-task rewards, with each λ_i serving as a scalar weighting coefficient.Source: C. Decomposing Multi-step Tasks with… · p. 4
  • For Pick-and-Place, the stages reward approach, successful contact and stable gripper closure, then transport toward the goal while avoiding collisions.Weights are set manually and held fixed; the paper applies RR only to Pick-and-Place, while Push and Slide are single-step tasks.Source: C. Decomposing Multi-step Tasks with… · p. 4

A. Simulation

Across PyBullet Panda tasks, first-layer transfer and reward redesign improve learning in many settings, but obstacle-aware Pick-and-Place is a notable exception for transfer. The experiments compare transfer strategies, reward redesign, and DRL baselines across obstacle-free and obstacle environments.

A. Simulation

  • Transferring all critic layers slowed convergence and lowered final success relative to scratch across tested tasks; deeper layers encode source-task reward and obstacle-specific value estimates.The proposed controlled strategy transfers only first-layer critic parameters, which provide a more generic state–action encoding.Source: A. Simulation · p. 5
  • Reach→Push→SlideObstacle reaches ∼0.8 in 0.4–0.6M steps and plateaus around 0.85–0.88, while scratch reaches ∼0.8 around 2.2–2.6M steps.PushObstacle transfer reaches ∼0.7 within ∼10^5 steps and plateaus around 0.88–0.90, compared with scratch approaching ∼0.7 after (5–7) × 10^5 steps and plateauing around 0.70–0.75.Source: A. Simulation · p. 5, Table III · p. 6
  • On Reach, reverse transfer reaches ∼0.8 around 4–5K steps versus 6–7K from scratch, while all methods reach near 1.0 by 12–15K steps.For ReachObstacle, transferred policies exceed ∼0.9 within 2.0–2.5K steps versus 2.8–3.0K for baseline, with all variants ultimately near 1.0.Source: A. Simulation · p. 5
  • 0.83 ± 0.05 on Push with obstacle and 0.95 ± 0.02 on Slide with obstacle; the framework exceeds all DRL baselines on Push and improves over SAC on Slide.Pick-and-Place with obstacle is the exception: transfer reaches 0.57 ± 0.25 versus TQC at 0.67±0.22.Source: A. Simulation · pp. 4–5, Table II · p. 6
  • Reach→PickObstacle lags scratch: both reach ∼0.7 success, at 1.5–1.6M and 1.3M steps respectively, and the final plateau favors scratch, ∼0.80 versus 0.76–0.78.The authors suggest a reaching prior may bias exploration toward collision-prone approach directions in obstacle-aware grasp-and-place sequences.Source: A. Simulation · p. 6
  • Reward redesign reaches ∼0.9 success on obstacle-free Pick-and-Place around 0.18–0.22M steps, versus 0.70–0.75M for the sparse baseline.With an obstacle, RR reaches ∼0.7 at 0.55–0.60M steps versus around 1.0M for the sparse baseline, but all variants plateau near 0.80–0.83 and TL+RR does not consistently improve performance.Source: A. Simulation · p. 6

B. Real-world experiment

The authors demonstrate a real-world pick-and-place task with an obstacle on a Franka Emika FR3. They select a simulation trajectory and adapt its execution to the physical robot.

B. Real-world experiment

  • A single best-performing trajectory is selected in PyBullet and executed open-loop on the Franka Emika FR3.The simulated setup replicates the real initial end-effector and object configurations, including geometry and spatial arrangements.Source: B. Real-world experiment · p. 6, Figure 5 · p. 7
  • A 7-DoF arm model aligns simulated joint-space representations with the FR3 despite differences in kinematic structure and degrees of freedom.Linear interpolation smooths trajectory segments with large joint displacements while meeting kinematic and safety constraints.Source: B. Real-world experiment · p. 6
  • Fig. 5 illustrates the real-world pick-and-place task with an obstacle on the FR3.The paper describes this deployment as assessing applicability to practical manipulation tasks.Source: B. Real-world experiment · p. 6, Figure 5 · p. 7

V. CONCLUSION AND FUTURE WORKS

The framework combines controlled transfer learning and reward redesign, improving obstacle-task performance and training efficiency while showing that transfer benefits depend on task structure and direction.

V. CONCLUSION AND FUTURE WORKS

  • Final success rates improve by up to 7.8% on obstacle-aware Push and 11% on obstacle-aware Slide, while reward redesign accelerates Pick-and-Place training by up to 73%.The framework’s gains vary by task: transfer is less helpful for obstacle Pick-and-Place, where reward redesign is more reliable.Source: §V · p. 7
  • Transfer learning improves convergence speed and final performance in Push and Slide, but obstacle Pick-and-Place shows no clear benefit from transfer alone because of geometric mismatch.Double transfer further improves performance through progressive structural alignment; reverse transfer does not degrade performance in simpler tasks.Source: §V · p. 7
  • Reward redesign is particularly effective for multi-step tasks such as Pick-and-Place, although its impact on final performance under strong geometric constraints is less consistent.In obstacle Pick-and-Place, it accelerates early learning and still speeds training when combined with transfer learning.Source: §V · pp. 7–8
  • The first-layer transfer protocol reuses general state-action representations while avoiding interference from task-specific value structures.Source: §V · p. 8
  • The framework was demonstrated on a real Franka Emika FR3 robot in obstacle-aware Pick-and-Place as a qualitative sim-to-real feasibility test.Source: §V · p. 8
  • Future work targets broader environmental variation, stronger statistical evaluation, and closed-loop real-robot trials across randomized runs.Planned extensions include more seeds with significance testing, varied obstacle and object geometries, and reporting real-world success rates, failures, and baselines.Source: §V · p. 8

Related papers

Loading 2610.35494v1…