Combined Constrained Sampling and Reinforcement Learning for Robotic Manipulation

1 Introduction

• RL can struggle to explore because informative rewards may require reaching rare contact states.Contact-mode switches also make direct model-based trajectory optimization challenging due to contact dynamics’ nonlinearities and discontinuities.Source: §1 · p. 1
• Existing reset strategies shape which states and contact modes policies practice, but do not generally specify diverse configurations needed for non-prehensile tasks such as balancing.Prior approaches rely on heuristics or task-specific mechanisms, including forward simulation, random scene generation, or grasp-specific samplers.Source: §1 · pp. 1–2
• The proposed constrained sampler complements a physics simulator by sampling starts and goals across diverse contact modes.The sampler defines distributions for goal-conditioned RL using constraints on manipulation states.Source: Figure 1 · p. 2, §1 · p. 2
• The approach combines constrained sampling with projected interpolation and curriculum learning, and experimentally validates improved performance for non-prehensile manipulation.The policies reach goals that can be held static and exhibit dynamic, robust behaviors using whole-body and environmental contacts.Source: §1 · p. 2

Model-based Manipulation Planning

• Model-based manipulation planning exploits environmental constraints and explicit [22] [5] or implicit contact-mode formulations, but contact dynamics challenge gradient-based solvers.Shooting methods [26] and RL [7] treat dynamics as black-box simulation, avoiding those solver challenges but not exploiting contact structure.Source: §2 · p. 2

Goal-conditioned RL & Universal Policies

• Goal-conditioned RL methods include reward relabeling [19] [31], shaping [32], subgoal generation [33] [34] [35] [36], curricula, and expert guidance [34].These approaches generally assume that start and goal states can be sampled as part of the problem definition.Source: §2 · p. 3

Goal and Reset Distributions in RL

• Prior reset methods expand visitation locally [14] [42] [16] or use clutter generation [15] and grasp samplers [17], with limited applicability to non-prehensile and whole-body manipulation.The paper instead constructs reset and goal distributions from general manipulation-state constraints to expose policies to diverse contact configurations.Source: §2 · p. 3

3.1 Problem Formulation

• The sampler defines starts and goals over a constrained state space, whose distribution must be sampled by solving a nonlinear constraint problem.The constrained space provides a state-visitation prior but is not assumed to include every state reachable under the black-box dynamics.Source: §3.1 · p. 3

3.2 State Sampler

• The state sampler imposes necessary collision, contact, force, and quasi-static equilibrium constraints, then samples contact modes and solves a proximal nonlinear program.It represents contact points and forces explicitly; a random box-constrained state is projected toward feasibility using an augmented Lagrangian method.Source: §3.2 · p. 4

3.3 Additional Reset Strategies

• Projected interpolation blends sampled start and goal embeddings, then resets to the nearest sampled state in the constrained space.Direct interpolation could violate constraints such as collision, so the interpolated state is projected approximately using a kd-tree in embedding space.Source: §3.3 · p. 5
• An optional curriculum initially resets closer to the goal and shortens rollouts, then increases the interpolation range until it matches projected interpolation.The phase variable α is scheduled from near 0 to 1 during an initial training phase.Source: §3.3 · p. 5

4.1 Evaluation Domains

• The four domains test quasi-static manipulation, whole-body contact skills, cube orientation control, and manipulation with a Panda arm.They comprise double-sphere, panda-sphere, sphere-cube, and panda-cube.Source: §4.1 · p. 5, §4 · p. 5

4.2 Comparison of Alternative State Samplers and Reset Strategies

• 54.7% for HER alone versus 92.3% for TD7 with CCi on panda-sphere; constrained goal and reset sampling also outperforms versions without it across RL engines.Table 2 evaluates alternative RL methods inside CSRL.Source: Table 2 · p. 6, §4.2 · p. 6
• Constrained starts and goals significantly outperform random sampling on double-sphere, panda-sphere, and sparse variants, where random resets may miss static inter-limb holds.On the simplest double-sphere domain, random resets can achieve considerable but sub-optimal success when goals come from the constrained sampler.Source: §4.2 · p. 6
• Projected interpolation and curriculum further improve performance across domains and RL methods; interpolation is crucial on sparse panda-sphere and may help with sparse rewards.The cube-domain evaluation also confirms benefits from interpolation and curriculum; those tasks required a 4M replay buffer and 300k training steps for stable training.Source: §4.2 · pp. 6–7, Table 3 · p. 7
• Projected interpolation outperforms reverse curricula on double-sphere and panda-sphere, including when reverse curricula use random-drop starts.Reverse curricula were explored only on small domains because generating reset states on the fly incurred substantial computational overhead.Source: Table 3 · p. 7, §4.2 · p. 7

4.3 Character of found behaviors

• Behaviors include balancing and perturbation recovery, whole-body sphere scooping, cube edge or corner balancing, and top-squeeze cube rotation, though some cube goals miss precision thresholds.The edge- and corner-balanced cube behaviors can be dynamic and reactive, but sometimes fall slightly short of the goal threshold.Source: §4.3 · pp. 7–8

4.4 Real-World Transfer

• Real-world transfer uses OptiTrack-tracked cube manipulation with a Panda and an end-effector-attached ball, treating the simulated sphere robot as transferable single-contact skills through inverse kinematics.The paper illustrates a pushing behavior in this setup.Source: §4.4 · p. 8

5 Conclusion

• The constrained sampler complements a physics simulator by encoding diverse manipulation states that may be reachable, while the simulator samples dynamics.The proposed reset strategy can be combined with interpolation and curriculum to train universal non-prehensile policies.Source: §5 · p. 8
• Evaluations demonstrate whole-body contact-rich manipulation in panda-sphere and agile cube balancing on edges or corners in sphere-cube.Source: §5 · p. 8
• Training assumes full state access, with no sensor-based student transfer addressed; complex-object constraints also make single-CPU NLP sampling costly.The authors expect manipulation-specific sim-to-real challenges and suggest parallelization and dedicated solvers as future work.Source: §5 · p. 8

A Gym Environment Details

• Cube goals encode pose with five scene points, while sphere goals use the sphere center; success is based on the L2 distance to the target.Sparse rewards are 1 on success; shaped rewards provide progress and have episode sums bounded as described.Source: A Gym Environment Details · p. 13
• Observations include robot and object state, control error, scene points, and goal error; the control error supports Markovian observations and indirect force information.For Panda scenarios, scene points also include three gripper markers.Source: A Gym Environment Details · p. 13
• Projected interpolation embeds weighted robot pose and goal features, using weights 0.5 for sphere domains and 0.2 for cube domains.The table reports environment-specific observation dimensions and training and control parameters.Source: A Gym Environment Details · p. 13, Table 4 · p. 13

B TD7 & HER Extension Details

• TD7 is the exchangeable RL method used; implementation changes include a 10k batch size and a 10^-3 training step size.Harder problems used replay buffers up to 4M samples.Source: B TD7 & HER Extension Details · p. 14
• Clipping Q-targets to at least zero stabilized training with non-sparse shaping rewards, whose returns could otherwise be negative for random actions.HER relabels completed episodes by selecting a random episode state as the new goal and updating observations, rewards, and termination flags.Source: B TD7 & HER Extension Details · p. 14

C Statistics of Constraint Sampling

• Sampling without the Panda robot is considerably faster, requiring fewer NLP queries per sample and less wall-clock time; replacing a sphere with a cube does not severely increase complexity.Samples are i.i.d., so parallelization can further reduce compute time in complex scenes.Source: C Statistics of Constraint Sampling · p. 14

D Real World Transfer Setup

• Real-world transfer uses a Franka Emika Panda with a custom ball end-effector and Optitrack tracking of the cube’s position.The ball has a 3 cm radius and sits 16 cm from the manipulator flange; tracking uses 12 Flex13 cameras.Source: D Real World Transfer Setup · p. 14, Figure 4 · p. 15

E Comparison to Behavior Cloning from Diverse Trajectory Data

• Behavior cloning from open-loop trajectories contributed little beyond providing interpolating states to RL, while trajectory generation incurred substantial computational cost.The authors report that the prior-version quantitative results were generally below the current paper’s policies, following improvements to RL hyperparameters.Source: E Comparison to Behavior Cloning from… · p. 15

E.1 Zero-Order Optimization of Open-Loop Trajectories

• The trajectory optimizer seeks an open-loop control and state trajectory from a sampled start to goal under deterministic dynamics.It parameterizes controls with a small number of B-spline control points and minimizes terminal feature distance without control costs; this parameterization implies smooth trajectories.Source: §E.1 · p. 15
• The optimization feature combines object and robot positions and velocities with continuous support-contact indicators, whose contact component supplies an implicit gradient toward goal contact modes.In the evaluations, the feature weights are 2 for object position, 1 for robot coordinates, and 0.1 for velocities and contact indicators.Source: §E.1 · p. 16
• CMA-ES solves the black-box trajectory optimization, and feasible trajectories are added until the target dataset count is reached.Start and goal pairs are sampled from Ds; feasibility is determined by terminal cost at most ϵ.Source: §E.1 · p. 16

E.2 Behavior Cloning from Trajectory Data

• Directly sampling random starts and goals can make rewards initially unlikely, causing slow or muted learning progress.The described reward is binary and indicates whether object distance to the goal is below a threshold.Source: §E.2 · p. 16
• The method adapts TD3+BC-style behavior-cloning regularization to the TD7 framework, with goal conditioning and a BC data batch.The BC dataset is compiled by translating spline pieces aligned to MDP time steps into actions; deterministic simulation reproduces the corresponding state trajectories.Source: §E.2 · p. 16
• Sampling episode starts from trajectory data strongly biases RL, leaving open whether BC loss itself is effective; biasing initialization proved crucial in the authors’ research.Source: §E.2 · p. 16

• The specific quantitative results discussed in this section are referred to the original version 4.Source: §E.3 · p. 17
• Reset method drastically affects training behavior and final performance, whereas adding a BC objective has minor impact.Source: §E.3 · p. 17
• The authors argue trajectories chiefly demonstrate relevant interpolating states rather than actions.Source: §E.3 · p. 17
• Final policy rollouts show reactive pushes and recovery, unlike spline-based open-loop trajectories described as more conceptual than executable.The authors connect this contrast to challenges in distilling optimization-based trajectory data into reactive policies [57] [58].Source: §E.3 · p. 17

• The constrained sampler generated diverse samples across four domains.Source: F Random Constrained Samples from our… · p. 17
• The double sphere and panda sphere figures show constrained samples from their respective domains.Source: Figure 5 · p. 18, Figure 6 · p. 19
• The sphere cube and panda cube figures show constrained samples from their respective domains.Source: Figure 7 · p. 20, Figure 8 · p. 21