Source-linked AI summary
Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes
Alex X. Lee, Coline Devin, Yuxiang Zhou, Thomas Lampe, Konstantinos Bousmalis, Jost Tobias Springenberg, Arunkumar Byravan, Abbas Abdolmaleki, Nimrod Gileadi, David Khosid, Claudio Fantacci, Jose Enrique Chen, Akhil Raju, Rae Jeong, Michael Neunert, Antoine Laurens, Stefano Saliceti, Federico Casarini, Martin Riedmiller, Raia Hadsell, Francesco Nori
TL;DR
Robotic stacking with diverse geometric objects requires strategies beyond grasping and placing, but prior work has mainly focused on cubes or limited object sets. The paper introduces a broad-object benchmark and learns vision-based policies through simulation RL, interactive imitation, and sim-to-real transfer. The policies demonstrate varied stacking skills, and offline RL improves them using data gathered by sim-to-real policies.
Problem
Method
The method trains simulation experts, interactively distills vision policies for sim-to-real transfer, and applies offline RL to robot-collected data.
Source: §1 · p. 2, §4.2 · p. 5Results
Takeaways & Limitations
Abstract
from arXiv · showhide
We study the problem of robotic stacking with objects of complex geometry. We propose a challenging and diverse set of such objects that was carefully designed to require strategies beyond a simple "pick-and-place" solution. Our method is a reinforcement learning (RL) approach combined with vision-based interactive policy distillation and simulation-to-reality transfer. Our learned policies can efficiently handle multiple object combinations in the real world and exhibit a large variety of stacking skills. In a large experimental study, we investigate what choices matter for learning such general vision-based agents in simulation, and what affects optimal transfer to the real robot. We then leverage data collected by such policies and improve upon them with offline RL. A video and a blog post of our work are provided as supplementary material.
1 Introduction
The paper addresses stacking strategies that go beyond grasping and placing, using a benchmark of diverse geometric objects to study vision-based real-world stacking. It investigates skill mastery and generalization, and reports that simulation-trained agents can bootstrap offline RL data collection efficiently.
1 Introduction
- The task requires reasoning about inter-object contact dynamics and control strategies beyond grasping and placing objects.The paper distinguishes stacking structures from manipulation tasks that do not require complex contacts between objects.Source: §1 · p. 1
- The RGB-Stacking benchmark contains 152 procedurally generated and 3D-printed rigid objects, addressing limited prior work on cube-like or narrowly selected shapes.The paper emphasizes its large object variety and extensive real-world evaluations.Source: §1 · p. 1
- The study evaluates mastery of five fixed object combinations and generalization from training objects to held-out objects.The five combinations were selected to test precision, balancing, and using the top object to flip the bottom object.Source: §1 · pp. 1–2
- Simulation-trained agents are the most efficient way to bootstrap offline RL data collection in both task settings.The contributions also report that offline RL improves performance with data from sim-to-real policies, whereas scripted-agent data performs worse than zero-shot sim-to-real.Source: §1 · p. 2
2 Related Work
Prior real-robot stacking work largely focused on cubes or a small set of known objects, while large-scale manipulation learning often emphasized grasping and short-horizon tasks. The paper positions diverse-object stacking as a more challenging, reproducible benchmark and combines simulation RL, interactive imitation, and offline RL for it.
Robotic Stacking
- Earlier vision-based real-robot stacking mainly studied cubes or a small set of known irregular objects, while RGB-Stacking covers thousands of combinations and evaluates more extensively.The paper contrasts its benchmark with prior work on six known stones evaluated in 11 episodes.Source: §2 · p. 2
Large-Scale Deep RL for Robot Manipulation
- Large-scale real-robot learning often targeted grasping with short horizons and limited dynamical effects, unlike 20 Hz velocity control over 400-step episodes.The paper notes that long-horizon learning from recorded data can suffer from wrongly estimated Q-values and that real-world processes can be difficult to reproduce.Source: §2 · p. 2
Imitation Learning and (Offline) Reinforcement Learning
- The method trains state-based experts with MPO, distills them through DAgger-style interactive imitation, uses domain randomization for transfer, and applies filtered or success-only behavior cloning offline.The paper reports that DAgger-style corrections significantly outperform behavior cloning or policy distillation for this vision-policy stage.Source: §2 · p. 3
3 The RGB-Stacking Benchmark
RGB-Stacking varies object geometry to change grasping and stacking affordances, creating tasks that demand strategies beyond simple pick-and-place. Its two tasks test a single vision policy on five fixed triplets and transfer to objects held out by deformation axis.
3 The RGB-Stacking Benchmark
- The benchmark varies grasp and stacking affordances to require behaviors beyond simple pick-and-place, while retaining automated learning and evaluation.The paper assesses affordances qualitatively and quantitatively, including through human teleoperators and a scripted agent in simulation.Source: §3 · p. 3
3.1 The RGB-Objects Family
- Four deformation axes transform a seed cube into varied shapes, and their combinations form the parametric RGB-Objects family.The axes change polygon shape, trapezoid shape, extrusion orientation, and object scaling; the final set samples eight objects along each axis and pairwise combination.Source: §3.1 · p. 3
3.2 RGB-Stacking Tasks
- The stacking tasks use red as the top object, blue as the bottom object, and green as a distractor that can also obstruct stacking.A successful stack requires the red and blue objects’ centroids to align vertically within thresholds.Source: §3.2 · p. 4
3.2.1 Skill Mastery on 5 Specific Triplets
- One vision-based agent is tested on five fixed triplets with distinct demands including grasp precision, flipping, balancing, centroid alignment, and preventing a near-cylindrical top object from rolling off.The triplets were selected to span different stacking difficulties; the figure shows all five stacks by one simulation-trained agent transferred zero-shot.Source: §3.2.1 · p. 4, Figure 3 · p. 4
3.2.2 Skill Generalization
- The generalization task trains on 103 shapes from pairwise-combined deformation axes and evaluates on 40 held-out shapes from four single-deformation axes.The five fixed triplets from the mastery task belong to the held-out axes.Source: §3.2.2 · p. 4
4 Stacking a Diverse Set of Objects in the Real World with a General Policy
The approach separates expert skill learning, vision-policy distillation, and real-robot policy improvement. It uses simulation-trained state experts, interactively distills a vision policy under domain randomization, then trains an improved policy offline from robot-collected data.
4 Stacking a Diverse Set of Objects in the Real World with a General Policy
- The pipeline trains experts from state information in simulation, distills vision policies under visual and dynamics domain randomization, and improves them with real-robot data.The paper says explicitly separating the stages allows faster, cheaper iteration through parallel experimentation and tuning.Source: §4 · p. 4
4.1 Training Expert Policies from State Features in Simulation
- Off-policy RL with shaped rewards trains state-based experts, using MPO for specialists conditioned on triplet identity or a generalist conditioned on object deformation parameters.Training directly from state was significantly faster than training from vision; the paper notes other RL algorithms could also be used.Source: §4.1 · p. 5
4.2 Interactive Imitation Learning of a Vision-Based Policy for Sim-to-Real Transfer
- Interactive imitation trains a vision student on its own visited states with teacher action corrections, addressing covariate shift that harms behavior cloning.Optimization runs alongside student data collection rather than alternating collection and supervised learning; mastery uses five triplet-specific teachers, while generalization uses one conditioned generalist.Source: §4.2 · p. 5
4.3 One-Step Offline Policy Improvement for Sim-to-Real Policies
- One-step offline improvement deploys the sim-to-real policy to collect robot data, trains an improved policy with a filtering function, and evaluates that policy on the robot.The paper considers CRR-IMP using a learned Q-function and BC-IMP on successful trajectories only.Source: §4.3 · p. 5
5 Experiments: Solving RGB-Stacking
The experiments test vision-based RGB stacking across fixed-triplet and held-out-object settings, comparing simulation learning, sim-to-real transfer, and offline improvement. Interactive imitation outperformed behavior cloning in simulation, while offline RL improved real-robot policies trained on sim-to-real-agent data.
5 Experiments: Solving RGB-Stacking
- Evaluation randomized object positions and measured whether the red object remained stacked on the blue at episode end, using 20-second episodes at 20 Hz.Real-world evaluations used 200 episodes per triplet and averaged success rates across triplets.Source: §5 · p. 6
5.1 Evaluation on the Simulation Stacking Benchmark
- State agents learned 10 times faster than vision agents, while interactive imitation significantly outperformed behavior cloning in both task settings.State-based MPO achieved 79.3% on Skill Mastery test triplets and 68.8% on Skill Generalization training objects, but only 47.8% on held-out test triplets.Source: §5.1 · p. 6
5.2 Evaluation on the Real-World Stacking Benchmark
- Binary gripper actions mattered most for sim-to-real transfer, followed by deterministic robot execution; Transformer attention helped exploit expert skill variety in generalization, but not mastery.The corresponding simulation performance changed less with the gripper-action choice.Source: Table 2 · p. 7, §5.2 · p. 7
- One offline step on sim-to-real data raised Skill Mastery from 69.6% average success to 74.6% (BC-IMP) and 81.6% (CRR-IMP).By contrast, scripted-agent data did not improve performance.Source: §5.2 · p. 8, Table 3 · p. 8
- Skill Generalization improved on held-out triplets using 38,446 training-object episodes, 37.4% successful, collected by an earlier agent with 32.6% test-triplet performance.Both BC-IMP and CRR-IMP improved performance using this data.Source: §5.2 · p. 8
6 Conclusion
The paper presents a benchmark for vision-based stacking of diverse geometric objects, reports progress on skill mastery and generalization, and identifies a remaining gap between those tasks. Its object family combines cube deformations to vary grasping and stacking affordances.
6 Conclusion
- The best agent is a single vision-based policy that performs varied stacking strategies, while a gap between Skill Mastery and Skill Generalization remains an open challenge.The benchmark is intended to support development of general policies, including possible adaptation at deployment.Source: §6 · p. 8
A The RGB-Stacking Benchmark
- The benchmark evaluates grasp and stacking affordances qualitatively and quantitatively, including through simulated human teleoperators and a scripted agent.It features diverse geometric objects and tens of thousands of possible stacks.Source: A The RGB-Stacking Benchmark · p. 14
A.1 The RGB-objects family
- The object family applies four major deformations to a seed cube and combines them into a parametric set, using pairwise combinations for training and single axes as held-out axes.The benchmark includes 103 training shapes and 40 held-out shapes; evaluation uses five fixed test triplets.Source: §A.1 · pp. 14–15
A.2 Benchmark Analysis
The benchmark varies grasping and stacking affordances through geometric deformations, and the robot setup combines a constrained Cartesian controller with vision and proprioception. Object tracking supports rewards, resets, and scripted baselines, but is not available to the learned agent.
A.2 Benchmark Analysis
- Object deformations vary grasp affordances: polygonal shapes require precise positioning, trapezoids demand positioning and orientation accuracy, and elongated rectangles can prevent some grasps.The average Ferrari-Canny epsilon grasp metric is used to quantify grasp difficulty; a lower value indicates harder grasping.Source: §A.2 · pp. 16–17, Figure S4 · p. 17
- Stacking affordances vary with support shape, area, and orientation; offset support can require careful placement to avoid tipping, while reduced support may increase rolling risk.Clutter restricts feasible grasps, and some valid grasps are unsuitable for stacking.Source: §A.2 · pp. 17–18, Figure S5 · p. 18, Figure S6 · p. 18
A.3 Publicly Released Objects
- The released objects use standardized manufacturing rather than standardized purchases, with single-color 3D-printed objects in red, green, or blue.The single-filament design requires no additional assembly steps.Source: §A.3 · p. 18
B.1 Real-World Environment
- The real-world system uses a Sawyer arm, a Robotiq gripper with custom nylon fingertips, force-torque sensing, cameras, and a confined basket workspace.The TCP is limited to a 20 cm-high virtual cube above a 25 cm × 25 cm basket surface.Source: §B.1 · pp. 18–19
B.1.1 Control Actions
- The controller exposes four Cartesian motion degrees of freedom—three translational and one vertical rotation—plus gripper motion, while a constrained least-squares solver computes joint velocities.Constraints enforce joint position and velocity limits and avoid collisions with the playground.Source: §B.1.1 · p. 20
B.2 Environment Observations
- The evaluation observation set removes tracker information, and observations are stacked over time to address actuation delays; camera images are excluded from stacking for real-time and memory reasons.The learned agent cannot access tracked Cartesian object positions, and omits the back-camera image to reduce inference time.Source: §B.2 · p. 20
B.3 Object Position Estimation
- Object tracking estimates colored-object centroids by masking color in YUV images, finding the largest contour per color, and triangulating across calibrated cameras.Tracking supports reward computation, environment resets, and the scripted baseline.Source: §B.3 · pp. 21–22
B.4 Task Evaluation
Evaluation uses randomized trials on real and simulated robots, with an explicit geometric and gripper-based definition of success. Automated procedures support repeated evaluation and data collection, while simulation training uses a staged reward and human teleoperator demonstrations provide a separate difficulty reference.
B.4 Task Evaluation
- Each real-robot trial lasts 20 seconds at 20 Hz, with randomized initial object and robot states; trials stop early when force thresholds are exceeded.Unless otherwise specified, performance is tested over 200 trials.Source: §B.4 · p. 22
B.4.1 Reward Definition
- A real-world stack requires the top-object centroid to lie within a 3 cm-radius cylinder at least 2.5 cm above the bottom object, with the gripper fully open.The open-ended cylinder accommodates objects of differing sizes.Source: §B.4.1 · p. 22
B.4.2 Automation for Unattended Learning and Evaluation
- Evaluation is largely automated across ten identical robot cells, with five fixed evaluation cells and camera adjustments used to improve comparability.Trials are repeated to reduce variance, and training from robot data is offline to some degree because evaluation requests are processed in order.Source: §B.4.2 · p. 23
B.5 Simulation
- Simulation matches the real setup’s observations and actions but differs in low-level appearance and object interaction physics; it also uses more episodes and full-pose randomization.Simulation evaluation uses 1,000 episodes per policy per triplet and privileged contact information unavailable on the real robot.Source: §B.5 · p. 23
B.5.1 Shaped Reward
- The simulation-only shaped reward progresses through reaching and grasping, lifting, hovering, stacking, and leaving the stack, giving incremental reward for task stages.The reaching term shapes distance between the TCP and top object over 15 cm; stacking rewards placement within 3 cm horizontally and 1 cm vertically of the target position.Source: §B.5.1 · pp. 23–26
C.1 Human performance
- Four people collected 846 teleoperated simulation episodes under the evaluation time limit, and these demonstrations were not used to train the reported agents.The object set was randomly replaced every ten episodes.Source: §C.1 · p. 26
C.2 Scripted Agent
The scripted baseline is a finite-state controller that uses object-centroid information and serves as a reference for task-solving without reasoning about object shape or orientation. It achieves 43% success on the training set in simulation.
C.2 Scripted Agent
- The baseline is a nine-state finite-state machine with transitions for grasping, lifting, moving above the bottom object, placing, and recovery when a state fails.If stacking is detected, execution transitions to the final state; failures route to a repositioning state that loops back to grasping.Source: §C.2 · p. 27
- The controller uses a low-gain position controller for TCP motion and randomizes wrist rotation during selected states to vary grasp orientation.It uses the available agent observations plus tracked object centroids.Source: §C.2 · pp. 26, 28
- 43% success was achieved by the scripted baseline on the training set over 10,000 simulation episodes.Its test-set performance is summarized across simulated and real environments in Table S4.Source: Table S4 · p. 28, §C.2 · p. 28
D.1 Details on Training Expert policies from State Features in Simulation
The simulation pipeline trains state-based MPO experts for fixed or general stacking tasks, then distills them into a vision-based policy through interactive imitation learning. The appendix details the expert-training choices and the data-collection setup for distillation.
D.1 Details on Training Expert policies from State Features in Simulation
- State-based experts use full simulation state and a shaped reward, because training directly from state features is significantly faster than training from vision.Experts specialize in each of five fixed triplets for Skill Mastery or cover 1,092,727 triplets formed from 103 training objects for Skill Generalization.Source: §D.1 · pp. 28, 30, Table S5 · p. 29
- MPO learns a Q-function and uses a KL-constrained policy update, with trust-region constraints limiting policy changes for stable learning.The expert policy is updated every 200 optimization steps; the implementation also uses a Retrace n-step target.Source: §D.1 · p. 30
D.2 Details on Interactive Imitation Learning for Sim-to-Real Transfer
- The state-based experts are distilled into one vision-based policy that uses images and proprioception, without object-position information.Velocity, force, and torque observations are excluded during simulation distillation; the resulting observations are intended to support zero-shot sim-to-real transfer.Source: Table S5 · p. 29, §D.2 · p. 30
- Distillation collects data from 1,000 domain-randomized simulation actors running the learner, then trains on replayed data using expert feedback in a DAgger-style procedure.The policy's effect on data collection is implicit; the sampling process is not differentiated.Source: §D.2 · pp. 30–31
D.3 Details on Training Improved Policies from Real Data
The paper improves vision-based policies using real-robot data and describes the randomization and image augmentation used for transfer. The methods combine filtered offline policy learning with simulation variation and image transformations.
D.3 Details on Training Improved Policies from Real Data
- Offline improvement trains a vision-based policy on real-robot data with velocity observations, using episode-success filtering for BC-IMP or an exponential advantage filter for CRR-IMP.CRR-IMP learns a distributional action-value function alongside the policy to estimate the advantage used by its filter.Source: §D.3 · p. 31
E.1 Domain Randomization and Image Augmentation
- To support transfer, the simulation is aligned with the real robot and uses domain randomization and image augmentation to address remaining domain gaps and real-world stochasticity.The authors report that preliminary learned-adaptation methods did not show clear benefits beyond randomization and augmentation.Source: §E.1 · p. 31
E.1.1 Domain Randomization
- Simulation randomizes physical properties and action delays, as well as visual properties; RGB-object colors remain within predefined hue ranges and are not assigned textures.Most physical properties vary uniformly within ±10% of default values, while gripper friction ranges were tuned to avoid unrealistic grasping.Source: §E.1.1 · pp. 31–32, Figure S16 · p. 32, Table S6 · p. 33
E.1.2 Image Augmentation
- Image augmentation applies brightness, hue, saturation, contrast, and translation transforms to both simulated and real images, using consistent random offsets across each trajectory subsequence.Hue perturbations are small enough to keep red, green, and blue objects reasonably close to their respective colors.Source: §E.1.2 · pp. 32–33, Table S7 · p. 33
E.2 Additional Network Architecture Details
The appendix specifies the vision- and state-agent networks, including temporal attention for vision-based control, and lists additional training choices. It also reports that shared image-encoder parameters improved transfer in preliminary experiments.
E.3 Additional Training Details
- State-based agents use an input normalizer, MLP, and output distribution layer, while training uses separate actor, critic, and dual-variable learning rates with actor and critic annealing.MPO and CRR use separate trust-region constraints for the mean and standard deviation of the continuous actor distribution, but not for its Bernoulli component.Source: §E.2 · p. 34, §E.3 · p. 36
E.2 Additional Network Architecture Details
- Vision-based agents combine image and proprioception encoders, followed by an MLP, Transformer, and output layer; the actor and critic use separate encoder parameters.Each image is processed by a ResNet stream, while proprioception is processed by an MLP before the encoded inputs are concatenated.Source: §E.2 · p. 34, Table S9 · p. 35
- Processing each image with a ResNet and sharing the two ResNets' parameters improved sim-to-real transfer in preliminary experiments, with parameter sharing also speeding computation.The comparison was against stacking both images along the channel dimension for one ResNet.Source: §E.2 · p. 34
- The Transformer encodes temporal information with repeated self-attention, multi-head projections, positional encoding, residual connections, layer normalization, and a feed-forward network.The paper motivates temporal input as a way to address transition-dynamics gaps in sim-to-real transfer.Source: §E.2 · pp. 35–36
E.4 Detailed Real-Robot Results
The detailed-results appendix reports individual-run outcomes and triplet-level results alongside the averaged results presented in the main tables. It identifies which supplementary tables correspond to the main results tables.
E.4 Detailed Real-Robot Results
- For each setting, Tables 2 and 3 report averages across four runs of sim-trained distilled agents: two seeds for each of the two state-based teacher policies distilled.Source: §E.4 · p. 36
- The appendix provides results for each run and each specific triplet to show the variance within each setting.Source: §E.4 · p. 37
- Supplementary Table S12 corresponds to Table 2, while Supplementary Table S13 corresponds to Table 3.Source: §E.4 · p. 37
E.5 Qualitative Analysis
The appendix qualitatively probes the best-performing Skill Mastery CRR-IMP policy under adversarial object placements, revealing geometry-specific skills alongside recovery gaps. It also situates the benchmark against related work and documents supplementary resources.
E.5 Qualitative Analysis
- Across Triplets 1 and 3, the agent waits for wrist alignment before grasping slanted sides and re-aligns an asymmetrically balanced top after an off-center grasp.Source: §E.5 · pp. 37, 39
- For Triplet 2, the agent tips the bottom object when its orientation lets it push against the basket slope, but does not rotate it from unfavorable orientations.Source: §E.5 · pp. 37, 39
- In Triplet 4 it usually centers the top object but does not recover from off-center stacks; in Triplet 5 it sometimes supports the rounded top with its gripper to prevent rolling.Source: §E.5 · p. 39
F Additional Related Work
- Furrer et al. evaluate six specific stones over 11 episodes, unlike this benchmark’s 152 diverse objects and more than 54,000 evaluated episodes.Other cited work addresses simulated or state-based cube stacking, high-level planning, or stack stability, rather than real-robot contact dynamics and perception [60] [61] [64] [67] [68] [69].Source: F Additional Related Work · pp. 39–40
G Additional Supplementary Material
- Supplementary materials include a video, real-cell and basket building resources, and STL files; the authors also planned to release additional assembly materials, object files, and a simulated environment.Source: G Additional Supplementary Material · p. 40
- Supplementary figures show real-world and simulated image sequences with and without image augmentations, including domain-randomized simulation; perturbations remain fixed within each sequence.Source: Figure S18 · p. 41, Figure S19 · p. 42, Figure S20 · p. 43