Source-linked AI summary
MDPO: Multi-constraint Dexterous Policy Optimization with Reinforcement Learning
Ao Zhou, Zongyou Yu, Ke Wu, Tao Xu, Junqiao Zhao, Wenchao Ding, Alois Knoll, Guang Chen
TL;DR
Dexterous manipulation policies must meet multiple safety requirements, but existing constrained RL methods for the domain mainly focus on cumulative costs. MDPO combines cumulative and state-wise constraints through policy improvement and joint projection; experiments report better constraint satisfaction with competitive task performance against baselines.
Problem
Existing constrained RL methods for dexterous manipulation mainly focus on cumulative costs, despite the need to regulate heterogeneous cumulative and state-wise safety requirements together.
Source: B. Constrained RL · p. 2Method
MDPO formulates cumulative and state-wise cost constraints and projects a performance-improved intermediate policy into their joint feasible region.
Source: §I · p. 1, §VI · p. 8Results
Across five dexterous manipulation tasks, MDPO achieves competitive or higher success rates and episode rewards while maintaining relatively low cumulative costs and state-wise violation rates.
Source: B. Comparison Experiments · p. 7Takeaways & Limitations
Jointly incorporating cumulative and state-wise constraints with independently adapted correction strengths supports stable safety satisfaction while preserving competitive manipulation performance.
Source: C. Ablation Experiments · p. 8
Abstract
from arXiv · showhide
Dexterous manipulation requires high-dimensional control policies to accomplish complex contact-rich tasks while simultaneously satisfying multiple safety requirements, such as joint limits, unsafe finger configurations, arm collisions, and unstable object interactions. Constrained reinforcement learning (RL) provides a principled framework for incorporating these requirements into dexterous manipulation policy learning. However, existing constrained RL methods for dexterous manipulation primarily focus on cumulative cost constraints and struggle to simultaneously handle multiple heterogeneous constraints during dexterous policy learning. In this work, we propose Multi-constraint Dexterous Policy Optimization (MDPO) for safe dexterous manipulation. We first formulate dexterous manipulation as a multi-constraint policy optimization problem involving both cumulative cost constraints and state-wise cost constraints. Based on this formulation, we integrate multi-constraint projection into the dexterous manipulation policy optimization process. Specifically, we employ constraint-specific policy projections to map the intermediate policy that maximizes task performance onto the feasible regions defined by the cumulative cost constraint and the state-wise cost constraint. Extensive experiments on ReDMan, including safe joint control, finger safety, and grasping tasks, demonstrate that MDPO achieves superior constraint satisfaction while maintaining competitive task performance compared with the baselines. Our code is available at this link.
I. INTRODUCTION
Dexterous manipulation requires high-dimensional coordination under cumulative and state-wise safety requirements, while existing constrained RL approaches have largely addressed cumulative costs alone. MDPO formulates both constraint types and projects an improved policy into their joint feasible region; ReDMan experiments report better constraint satisfaction with competitive manipulation performance.
I. INTRODUCTION
- Existing constrained RL for dexterous manipulation mainly handles cumulative costs, leaving simultaneous regulation of heterogeneous cumulative and state-wise safety requirements insufficiently addressed.The requirements are coupled: relaxing cumulative safety can increase instantaneous safety risks.Source: §I · p. 1
- MDPO formulates dexterous manipulation as a multi-constraint policy optimization problem and projects an intermediate performance-improving policy into the joint safe region.The projection addresses cumulative and state-wise cost constraints together.Source: §I · pp. 1–2
- ReDMan experiments on safe joint control, finger-level safety, and grasping report better constraint satisfaction with competitive manipulation performance versus baselines.The benchmark includes diverse dexterous manipulation scenarios with physical constraints.Source: §I · p. 2
A. RL for Dexterous Manipulation
- Prior dexterous RL work emphasized task performance, often using reward penalties or early termination for safety; ReDMan introduced safety costs but mainly a single cumulative constraint.The related-work discussion identifies multiple heterogeneous safety requirements as an unmet need.Source: A. RL for Dexterous Manipulation · p. 2
B. Constrained RL
Constrained RL methods for dexterous manipulation have mainly handled cumulative costs, although cumulative and per-step safety constraints can interact. The paper formulates both requirements, distinguishing episode-level cost limits from restrictions on safety-critical behavior at individual steps.
B. Constrained RL
- Projection-based methods separate performance improvement from safety correction, but most prior methods primarily target cumulative constraints.The review also lists Lagrangian, interior-point, CVaR-based, penalty-based, state-augmentation, and world-model-based approaches.Source: B. Constrained RL · p. 2
- Existing safe dexterous manipulation studies largely address single cumulative constraints; adding multiple penalty factors may destabilize optimization or make it overly conservative.Relaxing cumulative safety may improve task performance while increasing the likelihood of state-wise violations.Source: B. Constrained RL · p. 2
III. PRELIMINARIES
- Cumulative constraints bound long-term safety violations, whereas state-wise constraints restrict safety-critical behavior at individual interaction steps.The paper treats both as necessary because the constraints are coupled during dexterous manipulation.Source: §III · p. 3
IV. METHODOLOGY
MDPO reformulates the state-wise constraint into a tractable trajectory-level form, then uses a two-stage optimization framework and a sample-based first-order implementation.
IV. METHODOLOGY
- MDPO first reformulates the state-wise constraint into a tractable trajectory-level form.This is the first stage of the methodology overview.Source: §IV · p. 3
- The two-stage framework improves policy performance before jointly projecting the policy under multiple constraints.The overview names policy performance improvement and joint multi-constraint policy projection as its stages.Source: §IV · p. 3
- A sample-based first-order implementation is presented for practical policy optimization.The overview identifies this as the practical optimization implementation.Source: §IV · p. 3
A. State-wise Constraint Reformulation
MDPO turns state-wise safety into a trajectory-level constraint using accumulated cost increments, then improves task performance and jointly projects the policy to satisfy cumulative and state-wise safety requirements.
A. State-wise Constraint Reformulation
- Accumulated state-wise cost increments equal the maximum safety cost encountered so far, making the per-interaction risk limit a tractable trajectory-level constraint.The cumulative constraint limits total unsafe behavior over an episode; the state-wise constraint limits maximum risk during individual interactions.Source: A. State-wise Constraint Reformulation · p. 3
B. Policy Improvement and Joint Multi-constraint Projection
- MDPO first improves task performance, then jointly corrects cumulative and state-wise risks while keeping the resulting policy close to the intermediate policy.The second-stage KL term preserves the performance improvement from the first stage while reducing both safety risks.Source: B. Policy Improvement and Joint… · pp. 3–4
- The first stage uses a reward-oriented policy update and monitors KL divergence from the current policy to limit excessive policy changes.Safety costs do not directly participate in the first-stage actor objective.Source: B. Policy Improvement and Joint… · p. 4
- The cumulative and state-wise feasible regions may only partially overlap, so MDPO projects toward their intersection; if ¯D ≤(1 −γ)¯c, the state-wise region is contained within the cumulative region.Under that threshold condition, satisfying the state-wise constraint also satisfies the cumulative constraint.Source: B. Policy Improvement and Joint… · pp. 4–5, Figure 1 · p. 5
C. Practical Policy Optimization
MDPO’s practical optimization uses separate critics and safety statistics for cumulative and state-wise constraints, adapting their corrections independently during joint projection.
C. Practical Policy Optimization
- MDPO maintains reward, cumulative-cost, and state-wise cost-increment critics and estimates all three advantages using generalized advantage estimation.The cumulative and state-wise safety contributions are regulated independently in the subsequent projection stage.Source: C. Practical Policy Optimization · p. 5
- Cumulative safety is estimated by mean episode cost, while state-wise safety uses a high quantile of each episode’s maximum instantaneous cost.The implementation uses q = 0.95 and smooths both statistics with exponential moving averages.Source: C. Practical Policy Optimization · p. 5
- MDPO independently adapts cumulative and state-wise safety coefficients, strengthening correction as risk rises and attenuating it when safety margins are available.The cumulative target uses ρC = 0.9, while the state-wise target is ¯D.Source: C. Practical Policy Optimization · p. 5
- Fig. 2 compares five task rows across success rate, episode cost, cost rate, and episode reward, with means, standard deviations, and cumulative cost limits shown.The rows are Joint, Finger, Armjoint, Armfinger, and Grasp.Source: Figure 2 · p. 6
V. EXPERIMENTAL EVALUATION
The section evaluates MDPO’s effectiveness in multi-constraint safe dexterous manipulation.
V. EXPERIMENTAL EVALUATION
- The experiments evaluate the effectiveness of MDPO in multi-constraint safe dexterous manipulation.Source: §V · p. 6
A. Experimental Setup
The experimental setup evaluates MDPO on five ReDMan tasks, uses continuous safety costs, and adapts three constrained-RL baselines to the same multi-constraint setting.
A. Experimental Setup
- The evaluation covers Joint, Finger, Armjoint, Armfinger, and Grasp tasks spanning hand joints, fingers, arm-hand control, and object interaction.Source: A. Experimental Setup · p. 6
- The setup replaces binary safety costs with continuous costs while preserving task safety semantics, using them to construct cumulative and state-wise requirements.The running maximum enters the observation, and the state-wise cost increment provides an optimization signal; thresholds are ¯c = 35 and ¯D = 1.Source: A. Experimental Setup · p. 6
- LAG, P3O, and CRPO are adapted to incorporate both cumulative and state-wise constraints, and all methods use the same interaction budget and task settings.The adaptations use separate multipliers, penalty terms, or safety-oriented update decisions, respectively.Source: A. Experimental Setup · pp. 6–7
- The same implementation hyperparameters are used across all tasks.Source: A. Experimental Setup · p. 7
B. Comparison Experiments
Across five dexterous manipulation tasks, MDPO balances task performance with cumulative and state-wise safety. Its advantage is most evident on challenging arm-hand and grasping tasks.
B. Comparison Experiments
- Across five tasks, MDPO achieves competitive or higher success rates and episode rewards while maintaining relatively low cumulative costs and state-wise violation rates.Its advantage is more evident on ArmJoint, ArmFinger, and especially Grasp, where policy learning is more challenging.Source: B. Comparison Experiments · p. 7
- MDPO converges faster and more stably on most tasks, although several baselines eventually reach comparable performance on Joint and Finger.Those baselines generally require more interactions or show larger training fluctuations.Source: B. Comparison Experiments · p. 7
- Some baselines satisfy cumulative-cost limits yet retain notable state-wise violations, whereas MDPO jointly regulates long-horizon risks and instantaneous safety violations.Adaptive projection adjusts the two safety coefficients independently, while KL regularization limits deviation from the performance-improved intermediate policy.Source: B. Comparison Experiments · p. 7
C. Ablation Experiments
The ablations show that cumulative-cost and state-wise corrections address distinct safety risks, and neither alone achieves the best performance-safety trade-off. Their joint use supports more stable safety satisfaction while preserving competitive manipulation performance.
C. Ablation Experiments
- Neither cumulative-only nor state-wise-only safety regulation achieves the best performance-safety trade-off; MDPO combines both signals during policy projection.The cumulative constraint controls long-term accumulated violations, while the state-wise constraint suppresses instantaneous safety-critical behaviors.Source: C. Ablation Experiments · p. 7
- MDPO Cumulative regulates episode-level cost but generally has a higher state-wise cost rate, while MDPO State reduces state-wise violations but improves task performance more slowly and less stably.The ablation compares success rate, episode cost, cost rate, and episode reward on Joint and Finger.Source: C. Ablation Experiments · p. 7, Figure 3 · p. 7
- Jointly incorporating both constraints with independently adapted correction strengths yields more stable safety satisfaction while preserving competitive dexterous manipulation performance.Source: C. Ablation Experiments · p. 8
D. Real-world Deployment
MDPO’s learned grasping policy transfers to a physical robotic platform and successfully executes grasping with stable, safety-aware manipulation.
D. Real-world Deployment
- MDPO’s learned grasping policy transfers to a physical Tianji arm with a WujiHand dexterous hand and successfully executes the task.Source: D. Real-world Deployment · p. 8
- The grasping task requires coordinated arm-hand control while maintaining stable object interaction and avoiding aggressive or potentially unsafe hand motions.Source: D. Real-world Deployment · p. 8
- The physical deployment demonstrates the potential for transferring multi-constraint dexterous manipulation policies from simulation to real robotic systems.Source: D. Real-world Deployment · p. 8
VI. CONCLUSION
MDPO addresses cumulative and state-wise safety constraints through a two-stage optimization framework. ReDMan experiments show a favorable balance between task performance and multi-constraint safety.
VI. CONCLUSION
- MDPO achieves a favorable balance between task performance and multi-constraint safety in extensive ReDMan experiments.Source: §VI · p. 8
- MDPO reformulates the state-wise requirement as a tractable trajectory-level objective using a state-wise cost increment formulation.Source: §VI · p. 8
- Its two-stage framework improves task performance first, then projects the intermediate policy toward feasibility under both safety constraints.The joint projection uses independently adapted correction coefficients and KL regularization to preserve the intermediate policy’s manipulation capability.Source: §VI · p. 8