Soft-Bubble grippers for robust and perceptive manipulation

I. INTRODUCTION

• The Soft-bubble system combines a highly compliant gripping surface with dense-geometry visuotactile sensing for manipulation in tightly constrained environments.The introduction frames tactile perception as a response to post-grasp variability and unexpected disturbances in cluttered homes.Source: §I · p. 1
• The contributions include a smaller parallel-gripper form factor and internal markers for estimating shear-induced membrane displacement.The marker technique enables dense optical-flow tracking.Source: §I · pp. 1–2
• The system also introduces an optimization-based proximity pose-estimation framework inspired by depth-tracking methods.The introduction lists tactile classification using simultaneous images and an automated ground-truth labeling pipeline as another contribution.Source: §I · p. 2
• The authors demonstrate sensing effectiveness, robustness to uncertainty, and performance in human-robot interaction and antagonism tasks.The figure illustrates in-hand pose estimates and shear-induced displacements tracked inside the sensor.Source: Figure 1 · p. 1, §I · p. 2

A. Compliant Gripper Design

• Air-filled latex membranes combine pneumatic effects and membrane elasticity for compliance, while contact-patch friction supports grasping and the structure may resist wear.The paper contrasts this approach with compliant grippers based on solid materials and relates it to Kim et al.Source: A. Compliant Gripper Design · p. 2

B. Shear Estimation

• Uniform visual markers can produce shear-tracking errors on Soft-bubbles because their surfaces undergo large deformations and have an ellipsoidal form factor.The paper notes that Soft-bubble deformations on human-scale objects can reach multiple centimeters, versus multiple millimeters for GelSight or GelSlim.Source: B. Shear Estimation · p. 2
• A pseudorandom pattern-deposition technique enables dense optical-flow processing to track shear deformation.The related-work section frames compliant design and shear estimation as two of four areas addressed by the paper.Source: B. Shear Estimation · p. 2, §II · p. 2

C. In-hand Pose Estimation

• Depth registration methods such as ICP support local pose estimation but require a sufficiently close initial guess, whereas optimization-based methods such as DART are described as robust to initialization.The Soft-bubble deformations under a stable grasp capture enough object geometry for pose estimation from tactile information alone.Source: C. In-hand Pose Estimation · p. 2

D. Tactile Classification

• Tactile material and object classification uses a ResNet18 classifier on concatenated images from both sensors, with an automatic pipeline generating labeled images.The stated use case is recognizing a grasped object when other vision sensing cannot see it.Source: D. Tactile Classification · p. 2

III. MECHANICAL DESIGN

• The Soft-bubble fingers are designed to attach to a standard parallel gripper, handle human-scale objects, and fit tight household spaces.Perceptual design features include a shorter-range depth sensor, interchangeable bubble modules, and internal trackable patterns.Source: §III · pp. 2–3

A. Internal Depth Sensor

• The angled ToF sensor has a 4–11 cm working range and is positioned to maximize field of view and depth accuracy near finger contact regions while minimizing gripper width.Depth measurements are more accurate near the range’s upper bound.Source: A. Internal Depth Sensor · p. 3, Figure 3 · p. 3

B. Bubble Modules

• The bubble cartridge’s membrane, acrylic plate, mounting ring, inserts, and tube fitting are sealed together, while four screws allow the cartridge to be replaced.Modules can be exchanged to replace damage, change membrane thickness, or change internal patterning; the sensor electronics remain on the gripper.Source: B. Bubble Modules · p. 3, Figure 4 · p. 3

C. Printed Patterning

• Script-generated dot patterns are transferred to latex using laser-cut adhesive-backed stencils and screen-printing ink.The method supports single- or multiple-color patterns.Source: C. Printed Patterning · p. 3

D. Construction and Durability

• A batch of bubble-sensor fingers can be assembled in as little as two hours using accessible tools and off-the-shelf latex, without casting.The construction uses an FDM printer, laser cutter, scissors, glue, and a paint brush.Source: D. Construction and Durability · p. 3
• Experimental modules and stencils use varied pseudorandom internal dot patterns for shear tracking.The patterns include dots in both black and gray.Source: Figure 5 · p. 4
• Soft-bubbles operated for over 100 hours before the first failure, a puncture from a manipuland’s sharp edge.The puncture could be patched with negligible effect on tactile sensing capabilities.Source: D. Construction and Durability · p. 4

IV. PROXIMITY POSE ESTIMATION

• The estimator targets known-object pose from sensor measurements when the object is held in a stable, non-slipping grasp.Its design addresses poor or absent initialization in cluttered manipulation.Source: §IV · p. 4, Figure 6 · p. 4
• Proximity fields extend signed-distance fields with smooth gradients outside geometries, including at sharp corners and edges.For cylinders, the field combines nearest-corner and nearest-surface distances.Source: §IV · pp. 4–5
• The estimator optimizes a pose against concatenated sensor point clouds, with contact-patch filtering improving optimization speed and accuracy.The filtered cloud is typically around 10K points, and pose estimation runs at around 0.5Hz–1Hz.Source: §IV · pp. 4–5, Figure 8 · p. 5

• A pseudorandom interior dot pattern enables extraction of tangential displacement orthogonal to the applied gripper force.Source: §V · p. 5
• Dense optical flow compares a stable-grasp reference image with the latest camera image to estimate each image patch’s displacement.Source: §V · p. 5
• Summing the per-pixel flow vectors yields tangential displacement s, which is represented as the normalized direction s/∥s∥.Source: §V · p. 6

• The classifier can use either depth or infrared streams from both Soft-bubble cameras, each at 224 × 176 resolution.Source: §VI · p. 6
• Only stable grasps, identified using bubble-pressure differential and finger-velocity thresholds, are used for training and inference.Source: §VI · p. 6
• Images from both bubbles are concatenated and down-sampled to 224 x 224 for a ResNet18 classifier implemented in PyTorch.Source: §VI · p. 6

A. Automatic Labelling Pipeline

• Robot-generated training grasps address the need to represent diverse stable poses and object features such as mug handles and bottle caps.Human-generated grasp configurations yielded insufficient examples of geometries in robot-relevant poses.Source: A. Automatic Labelling Pipeline · p. 6
• About one minute of data, or 600 training samples per object, was sufficient for training once a stable grasp was obtained.Source: A. Automatic Labelling Pipeline · p. 6
• The pipeline generated 50 grasps (30K training samples) in 1.5 hours and reached ≥99% validation accuracy across three classes.Training converged in about 45 epochs; more rigorous analysis of the number of required grasps was out of scope.Source: A. Automatic Labelling Pipeline · p. 6

VII. EXPERIMENTS

• The system was deployed in a realistic kitchen with counters, a sink, and a dishwasher to demonstrate manipulation in a cluttered indoor scenario.Source: §VII · p. 6

A. Robot and Task Setup

• Bubble fingers on a Schunk WSG 50 gripper were attached to a KUKA iiwa arm, with six RealSense cameras covering the sink, countertop, racks, and dishwasher door.Plastic mugs, PET bottles, and wine glasses were used; OptiTrack validated in-hand pose estimation.Source: A. Robot and Task Setup · p. 6
• RealSense depth images were fused into a sink-scene point cloud, and grasp poses came from antipodal generation or a naive blob-centroid location.Source: A. Robot and Task Setup · p. 6
• Planning combined sampling-based and optimization-based methods, while parameterized closed-loop primitives scripted the robot’s actions.Perception, planning, and control ran on one Intel Xeon workstation under Ubuntu 18.04.Source: A. Robot and Task Setup · p. 6

VII. EXPERIMENTS

• Figure 10 shows four dish-loading grasp examples whose object classes were each identified with probability p > 0.95.Figure 9 illustrates a plastic-mug training sample using concatenated depth and infrared images.Source: Figure 9 · p. 6, Figure 10 · p. 7

B. Task 1: Dish loading with recycling separation

• The robot sorts grasped sink objects into valid dishes for dishwasher placement or recycling for bin disposal.The tested object selection was limited by the existing dish-loading pipeline, not by a stated fundamental limitation of Soft-bubbles.Source: B. Task 1: Dish loading with recycling… · p. 7

C. Task 2: Pose Estimation Under Antagonism

• Pose estimates for cylindrical mugs converged from seed offsets up to ±30 degrees in pitch and roll, but reliability is limited to cylindrical-surface contact.Stable grasps were initialized from a vision estimate when available or a cardinal pose, then tracked continuously from the previous estimate.Source: C. Task 2: Pose Estimation Under… · p. 7

D. Task 3: Shear-based Manipuland Release and Handover

• A closed-loop controller compares shear forces against a threshold to automate manipuland placement or handover.The threshold is relative to an initial stable grasp.Source: D. Task 3: Shear-based Manipuland… · p. 7

C. Task 2: Pose Estimation Under Antagonism

• The in-hand pose estimator runs at 1–2 Hz and tracks mug perturbations across grasp poses.The figure shows an upside-down mug and a perturbed grasp.Source: Figure 11 · p. 7

D. Task 3: Shear-based Manipuland Release and Handover

• The shear-based method was demonstrated handing a liquid-filled wine glass to a human and blindly stacking wine glasses without object or surface geometry knowledge.During handover, relative shear increases as the human takes the glass; release follows a threshold on shear magnitude.Source: D. Task 3: Shear-based Manipuland… · p. 7
• Shear thresholds depend on the post-grasp contact patch’s size and geometry.Future work proposes using contact-patch estimates and pressure data to improve shear sensing accuracy for more diverse objects.Source: D. Task 3: Shear-based Manipuland… · p. 7

VIII. CONCLUSIONS AND DISCUSSION

• Across real-world tasks, the Soft-bubble demonstrated in-hand pose estimation, shear-deformation estimation, and tactile classification within a robust manipulation pipeline.The system combines a highly compliant gripping surface and visuotactile sensing in a parallel-gripper form factor.Source: §VIII · p. 7
• The current implementation is limited in size by the minimum range of custom ToF sensors, which are difficult to source.The authors are working on customized ToF cameras for short-range depth sensing.Source: §VIII · pp. 7–8
• The shear tracker uses IR visual features, while planned algorithmic work includes incorporating depth and extending pose tracking to other geometric primitives.The authors hope depth-channel use will make the Soft-bubble more like a low-cost force-torque sensor; frustums are given as an example primitive.Source: §VIII · p. 8
• The authors identify tool use and physical human-robot interaction as future application areas.The handover figure depicts a human taking a liquid-filled wine glass as relative shear increases from the initial configuration.Source: §VIII · p. 8, Figure 12 · p. 8