PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Based 6-DoF Grasp Pose Estimation for Robot Cloth Unfolding

Domen Tabernik, Peter Nimac, Jan Jerićević, Danijel Skočaj, Andrej Gams

arXiv:2609.31452v1cs.ROcs.CV

TL;DR

Cloth’s deformability and visual ambiguity make reliable grasping for unfolding challenging. CeDiRNet-6DoF predicts grasp points and full 6-DoF poses from cloth images, and its evaluation reports gains on the combined set and competition performance. The authors identify scope limits in validation and per-location uncertainty prediction.

  • Problem

    Cloth’s high-dimensional deformability, complex dynamics, and occluded visual cues make grasping and unfolding difficult to automate.

    Source: §I · p. 1

  • Method

    CeDiRNet-6DoF uses dense grasp-location and orientation regression with segmentation to predict full 6-DoF cloth grasp poses for in-air regrasping.

    Source: §I · p. 1, B. Grasp Pose Estimation in 6-DoF · p. 3

  • Results

    17.2 pp. for coverage and 38.2 pp. for grasp success were the reported gains on the combined evaluation set.

    Source: §V · p. 12

  • Takeaways & Limitations

    The authors conclude that the framework supports cloth unfolding and generalizes to diverse cloth types and configurations in successful cases.

    Source: §V · p. 12, §VI · p. 12

    Validation covered towels and T-shirts, and the method predicts one pose per grasp location without calibrated uncertainty or multiple pose hypotheses.

    Source: §VI · p. 13

Abstract

from arXiv · show

Cloth manipulation is a challenging task due to the deformable and high-dimensional nature of cloth, which leads to complex interaction dynamics and perceptual ambiguity arising from frequent occlusions of critical visual cues such as folds, edges, and grasp points. In this work, we tackle cloth unfolding using a regrasping-in-the-air strategy, where one manipulator holds the cloth while the other grasps it at an optimally selected point to unfold it. To this end, we propose CeDiRNet-6DoF, a deep learning framework that jointly predicts effective grasp points and the complete 6-DoF grasp pose from the observed cloth configuration. By integrating dense 3D grasp regression with segmentation and sine-cosine-encoded Euler angles, the proposed method reliably estimates the grasp configuration that maximizes the unfolded cloth area. We extensively evaluated CeDiRNet-6DoF on a bimanual robotic setup within the ICRA 2024 Cloth Competition framework, achieving state-of-the-art performance. An ablation study further validates the benefits of key design components, including joint segmentation, background randomization, and image cropping. These results establish CeDiRNet-6DoF as a robust and versatile foundation for reliable robotic cloth manipulation in unstructured environments.

I. INTRODUCTION

Cloth’s deformability, nonlinear dynamics, and variable appearance make grasping and unfolding difficult to automate. This work combines learned grasp-pose prediction with in-air regrasping, addressing prior methods that focus on grasp points or broader manipulation policies.

I. INTRODUCTION

  • Cloth’s high degrees of freedom, nonlinear dynamics, and variable shape and appearance complicate manipulation in unstructured environments.These properties distinguish cloth from rigid objects with more predictable state changes under grasping or pushing.Source: §I · p. 1
  • The method adopts in-air regrasping: one arm suspends the cloth while the other selects a grasp intended to achieve full unfolding.The paper focuses on unfolding as a precursor to effective folding.Source: §I · p. 1
  • CeDiRNet-6DoF predicts grasp locations and full 6-DoF poses from a single RGB image for in-air cloth unfolding.It is evaluated under the ICRA 2024 Cloth Competition framework and includes an ablation study of design choices.Source: §I · pp. 1–2

A. Cloth Grasp Selection and Manipulation

  • Learning-based cloth methods detect grasp points or regions, while broader bimanual systems often predict coordinated grasp pairs or task-level policies rather than one 6-DoF unfolding grasp.Prior hand-crafted corner and edge cues can be less robust to self-occlusion, folds, and appearance variation.Source: A. Cloth Grasp Selection and… · p. 2

B. In-air Cloth Grasping and Direct Grasp Prediction

Prior cloth and rigid-object methods differ in sensing assumptions, target representations, and task scope. CeDiRNet-6DoF is positioned as single-view, direct in-air 6-DoF grasp prediction for the ICRA 2024 real-robot unfolding benchmark.

B. In-air Cloth Grasping and Direct Grasp Prediction

  • Tabletop cloth methods use task metrics, preference or reward guidance, or dynamic primitives, but do not address direct in-air 6-DoF grasp prediction.Related in-air methods may assume tactile sensing, multiple viewpoints, or extensive simulation training.Source: B. In-air Cloth Grasping and Direct… · p. 2

C. 6-DoF Pose Estimation and Grasping for Rigid Objects

  • Rigid-object pose methods rely on reference shapes or canonical frames, assumptions that do not map reliably to cloth’s continuously changing configurations.Rigid-object grasp definitions may also depend on stable geometry or whole-object grasping rather than a locally suitable cloth region.Source: C. 6-DoF Pose Estimation and Grasping… · p. 2

D. Benchmarks and Datasets

  • The ICRA 2024 Cloth Competition supplies a training dataset, execution setup, and evaluation protocol for end-to-end unfolding with in-air regrasping.It provides a full 6-DoF evaluation benchmark for the task.Source: D. Benchmarks and Datasets · p. 3

E. Positioning of CeDiRNet-6DoF

  • CeDiRNet-6DoF targets direct in-air 6-DoF grasp prediction from a single visual observation, rather than tabletop grasps, tactile or multi-view approaches, or broader coordinated policies.Its benchmark-aligned output is a single executable grasp pose.Source: E. Positioning of CeDiRNet-6DoF · p. 3
  • CeDiRNet-6DoF extends dense point localization to full pose prediction in a real-robot pipeline, using RGB for grasp estimation and depth for 3D positioning.Its adaptations include orientation regression, segmentation supervision, background randomization, and robot-aware cropping; it avoids rigid-object assumptions and learns from single grasp-point annotations.Source: E. Positioning of CeDiRNet-6DoF · p. 3

III. METHOD

CeDiRNet-6DoF extends dense point detection to predict grasp orientation and integrate the result into a robotic unfolding pipeline. Its dense fields support multiple grasp candidates, while camera depth supplies the 3D position.

A. CeDiRNet

  • CeDiRNet detects points through dense center-direction fields followed by LocNet localization, whose local maxima yield predicted target locations.The dense network may also predict log-distance to the closest target.Source: A. CeDiRNet · p. 3

B. Grasp Pose Estimation in 6-DoF

  • CeDiRNet-6DoF adds three Euler-angle orientation heads and cloth segmentation to the dense regression network.Each orientation head predicts a sine/cosine pair for roll, pitch, or yaw relative to the camera frame.Source: B. Grasp Pose Estimation in 6-DoF · p. 3
  • Multiple local maxima produce candidate grasp locations, each paired with a regressed 6-DoF pose for kinematic-feasibility-aware selection.The pose combines the localized 2D point and its predicted angles, with camera depth recovering the 3D position.Source: B. Grasp Pose Estimation in 6-DoF · pp. 3–4
  • Sine/cosine angle regression maps values into [-1, 1] and removes discontinuities at 2π, while Euler angles remain the practical representation used for the benchmark annotations.Euler angles still have axis-coupling and singularity-sensitivity limitations; alternative representations are left for future work.Source: B. Grasp Pose Estimation in 6-DoF · p. 4

C. Data Input

The input is cropped to the robot workspace, with RGB as the default and an object-centric depth encoding in RGB-D variants. Training jointly supervises dense outputs using balanced losses and auxiliary segmentation and distance tasks.

C. Data Input

  • Workspace-based cropping suppresses background and focuses the input on cloth details, with the same crop used in training and inference.The upper boundary is set 5 cm above the holding gripper; other boundaries use robot-base positions.Source: C. Data Input · p. 4
  • RGB is the default input; RGB-D variants add depth to the input layer using distance to a world XZ-plane, normalized from [-50, 50] cm to [-1, 1].This object-centric encoding avoids absolute camera depth and reduces camera-position and viewpoint dependence.Source: C. Data Input · p. 4

D. Learning

  • Training jointly supervises a shared head for center directions, log-distance, and segmentation, alongside three orientation heads for the Euler axes.Encoder and decoder weights are shared except for final dense output heads; uncertainty weighting balances the individual losses with learnable parameters.Source: D. Learning · p. 4
  • Segmentation and log-distance are auxiliary training outputs and are not needed at inference.Regression losses apply only to foreground regions around grasp points during training; inference predicts across pixels and reads orientation at detected grasp points.Source: D. Learning · p. 4

E. Inference

CeDiRNet6DoF produces grasp-pose candidates from cropped RGB or RGB-D images, then planning filters and ranks them to select an executable grasp. If none is feasible, the system repositions the cloth and retries.

E. Inference

  • DenseRegNet predicts center directions and three orientation fields; LocNet extracts 2D candidates, and camera depth supplies each candidate’s 3D position.Segmentation and log-distance outputs are auxiliary training outputs, while roll, pitch, and yaw are read at the detected grasp point.Source: E. Inference · p. 5

F. Planning and Grasping

  • Candidates are ranked by confidence and filtered for reachability, valid inverse kinematics, and trajectory feasibility before the highest-confidence executable grasp is selected.The target pose is transformed from camera to robot coordinates; redundant joint solutions are resolved using desirable configurations, and RRT-Connect plans a collision-free path.Source: F. Planning and Grasping · p. 5
  • If no candidate has a valid joint solution or is collision-free, the system repositions the cloth and reevaluates grasp candidates.It tries predefined holding-arm configurations; if no grasp is found for the initial cloth position, the attempt is considered failed.Source: F. Planning and Grasping · p. 5
  • The integrated pipeline is intended to translate estimated poses into safe, executable robot motions.The described sequence includes candidate ranking, feasibility checks, and trajectory planning.Source: F. Planning and Grasping · p. 5

IV. EXPERIMENTS

The experiments evaluate the dual-arm unfolding pipeline under the ICRA 2024 Cloth Competition task. The autonomous decision is the second grasp pose, and success is assessed by the cloth area achieved after stretching.

IV. EXPERIMENTS

  • The evaluated pipeline addresses dual-arm cloth unfolding using the regrasping-in-the-air strategy.The experiments assess the whole pipeline for this task.Source: §IV · p. 5

A. Task

  • The task lifts a cloth, uses its lowest hanging point for the first grasp, then requires an autonomous second grasp before both arms stretch it in the air.The first three task steps are predefined except for the second-grasp selection; the procedure follows the competition’s left-arm-holds, right-arm-grasps arrangement.Source: A. Task · p. 6
  • Success is measured by comparing the stretched cloth’s surface area with a reference area from a manually defined optimal stretch configuration.The reference maximizes cloth surface coverage.Source: A. Task · p. 6

B. Robot Setup

The experiments use a dual-arm setup modeled on the ICRA competition while training on towel data for pretraining and competition data for 6-DoF fine-tuning.

B. Robot Setup

  • The robot setup replicates the competition’s dual-arm arrangement but uses two 7-DoF Franka FR3 robots and basic hand grippers instead of UR5e robots and Robotiq grippers.The robots are positioned 90 cm apart, with a Zed2i RGB-D camera mounted behind them.Source: B. Robot Setup · p. 6
  • Because the FR3 and UR5e have different joint ranges and workspace shapes, the setup includes repositioning the cloth closer to or farther from the grasping arm.Their reaches are similar, but the FR3 has seven degrees of freedom and the UR5e six.Source: B. Robot Setup · p. 6

C. Training Datasets

  • Training combines ViCoS Towel data for pretraining with the organizer-provided ICRA 2024 competition dataset for task-specific fine-tuning.The ViCoS data includes 8,000 real and 12,000 synthetic images, plus 559 additional samples; the competition dataset contains 503 annotated samples from 28 towels and T-shirts.Source: C. Training Datasets · pp. 6–7

D. Training and Inference Details

The model is pretrained on towel grasp data and then fine-tuned for 6-DoF estimation on competition data. Evaluation compares models across cloth items and repeated trials using coverage and grasp success measures.

D. Training and Inference Details

  • Training proceeds in two stages: 3-DoF cloth-corner pretraining followed by fine-tuning the full 6-DoF model on the competition training split.The extended ViCoS Towel Dataset supplies pretraining data; the additional Euler-angle heads and segmentation output are initialized during fine-tuning.Source: D. Training and Inference Details · p. 7
  • 83.3% F1 on the ViCoS Towel 3-DoF benchmark followed the modified pretraining protocol and additional 559 images, compared with 78.4% previously.The protocol changes included a larger epoch size, image resizing, and replacing Gaussian blur with flips and rotations.Source: D. Training and Inference Details · p. 7

E. Evaluation Protocol

  • Table I reports coverage, grasp success, and coverage for successful grasps across the ICRA 2024 and independently collected sets.The table combines Runs A and B and marks items observed during training.Source: Table I · p. 7
  • The evaluation covered 16 items, with each model evaluated five times; eight models, seven unique, produced 640 grasping trials.The evaluation items comprised competition-provided and independently obtained cloths; 12 of 16 had not been observed during training.Source: E. Evaluation Protocol · pp. 7–8
  • Cloth coverage is the stretched cloth’s surface area divided by a reference area, and the evaluation also records successful grasps.Surface area is measured from cloth pixels in stereo-camera images, with the two coverage measurements averaged.Source: E. Evaluation Protocol · p. 8

F. Results

CeDiRNet-6DoF achieves strong grasping and coverage results on both evaluation sets, with variation across individual items. Its image-processing runtime is driven mainly by input resolution, not scene complexity.

F. Results

  • 61.3% average coverage and 72.5% grasp success on the ICRA 2024 set; on our set, the model achieved 57.4% average coverage and 62.5% grasp success.Coverage includes successful and unsuccessful grasps; for successful grasps, coverage was 72.6% on ICRA 2024 and 70.4% on our set.Source: F. Results · p. 8
  • 75.2% coverage and 80% grasp success on tshirt-women-airo contrasted with 35.9% coverage and 30% grasp success on cloth-checkered-green.The sleeveless tshirt-women-airo was not seen during training; failures on cloth-checkered-green often involved poor grasp orientation despite a suitable location.Source: F. Results · p. 8
  • 27 of 52 failed grasps involved incorrect orientation, while successful grasps often exceeded 80% or even 90% coverage.Performance was fairly consistent for shirt-pink, cloth-checkered-green, and tshirt-beige, but variance was larger for other items.Source: F. Results · p. 8

G. Computational cost

  • 0.27 to 0.81 sec on the A100 and 0.45 to 1.45 sec on the RTX 2080 Ti as resolution increased from 512×512 to 3840×2160.At 1024×1024, scenes with 5 versus 12 final detections both took 0.30 sec on the A100; scene complexity had negligible runtime effect.Source: G. Computational cost · pp. 8–9

H. Ablation Study

The ablations compare segmentation, background randomization, cropping, depth input and orientation regression. The final 6-DoF design outperforms the baseline, with the largest gains associated with random-background augmentation and robot-position cropping.

a) CeDiRNet-6DoF components

  • 60.9% combined-set coverage and 68.8% grasp success for the final model, versus 43.7% and 30.6% for the base model.Adding segmentation alone raised coverage to 47.4% and grasp success to 40.0%; the combined final-model results are shown in the ablation figure.Source: H. Ablation Study · pp. 9–10, Figure 6 · p. 11
  • 59.1% combined-set coverage and 65.0% grasp success with random-background augmentation and robot-position cropping, versus 43.7% and 30.6% for the base model.These were the largest gains among the tested individual components and are visible in Fig. 6.Source: H. Ablation Study · p. 10, Figure 6 · p. 11

b) Depth

  • 56.4% coverage with matched RGB-D variants versus 58.2% without depth, while grasp success remained 66.3%; the difference was not statistically significant.The authors therefore cannot conclude that depth input degrades performance.Source: H. Ablation Study · p. 10

c) Depth encodings

  • HHA scored below the no-depth and other encodings, while raw distances and surface normals had overlapping 95% credible intervals with the initial encoding and no-depth model.The overlapping intervals do not support a strong conclusion that raw distances or surface normals improve performance.Source: H. Ablation Study · p. 10, Figure 7 · p. 11

d) Baseline CeDiRNet-3DoF

  • 38.3 ± 3.5% 95% CI for the 3-DoF baseline versus 58.2 ± 5.5% 95% CI for CeDiRNet-6DoF, a statistically significant difference at the 0.05 level.The comparison used the 3-DoF baseline in the 6-DoF setup with yaw-only rotation regression.Source: H. Ablation Study · pp. 10–11

I. Comparing with State-of-the-Art

CeDiRNet-6DoF placed second in the ICRA 2024 cloth competition, outperforming nine other methods. In a human-judge proxy comparison, it also scored above two related-work baselines.

a) ICRA 2024 cloth competition

  • The competition entry used segmentation, random-background augmentation, robot-aware cropping, and RGB input; its setup differed mainly in robot arms and repetition count.The competition used two repetitions, compared with ten in the authors’ combined RUNS A and B.Source: §I · p. 11
  • 57% coverage placed CeDiRNet-6DoF second in the ICRA 2024 competition, behind AIR-JNU at 60% and ahead of nine other methods.Without two technically disqualified samples, its competition coverage would have been 59.01%, 0.99 percentage points below the first-ranked result.Source: §I · p. 11

b) Human judges evaluation

  • 1715.0 ELO-like score for CeDiRNet-6DoF versus 1455.2 for SAM3+EconomicGrasp and 1329.8 for EWHA-glab.The comparison used human judges and the same 100 evaluation samples as the prior proxy benchmark.Source: Figure 8 · p. 12, §I · p. 12

V. DISCUSSION

CeDiRNet-6DoF shows gains on cloth-unfolding evaluations, while its generalization evidence is limited to towels and T-shirts. The discussion and conclusion identify unresolved questions about depth, orientation prediction, uncertainty, and evaluation scope.

V. DISCUSSION

  • 17.2 pp. coverage and 38.2 pp. grasp-success gains on the combined evaluation set accompanied second place in the ICRA 2024 competition and the highest score in a three-method proxy comparison.The discussion also reports that CeDiRNet-6DoF outperformed EconomicGrasp, supporting its center-direction vectors and direct dense grasp-prediction formulation.Source: §V · p. 12
  • The evaluation supports generalization to unseen towels and T-shirts, but other garment categories remain unevaluated and require additional training data and dedicated evaluation.Novel T-shirts performed better than novel cloths; the authors suggest appearance differences may explain this, and note that checkered patterns may not have transferred to orientation prediction.Source: §V · p. 12, §VI · p. 13
  • The segmentation head is not used at inference, and its training loss may provide useful shared-feature supervision, although this mechanism is unconfirmed.The authors describe the regularization-like explanation as plausible rather than confirmed.Source: §V · p. 12
  • Depth did not benefit this model; the authors suggest the roughly 500-image fine-tuning set may overfit the depth channel of a backbone with more than 88 million parameters.Depth augmentation was proposed but not evaluated; RGB cues and pretraining may already provide useful information.Source: §V · p. 12

VI. CONCLUSION

  • The framework estimates full 6-DoF grasp poses and places them in a dual-arm pipeline using analytical inverse kinematics and RRT-Connect planning.The system follows a regrasping-in-the-air strategy for cloth manipulation.Source: §VI · p. 12
  • The evaluation covered eight original competition items and eight previously unseen items, with performance described as comparable to state of the art; ablations supported the design choices.The ablations found a slight benefit from segmentation-related feature adjustment, a benefit from a three-dimensional rather than single-dimensional orientation loss, and significant contributions from background randomization and robot-position-based cropping.Source: §VI · p. 12
  • The predictor selects among grasp candidates but produces only one pose per candidate, without calibrated uncertainty or multiple pose hypotheses; the benchmark likewise calls for one final grasp.The authors identify more diverse environmental variation and improved orientation prediction as future directions.Source: §VI · p. 13

Related papers

Loading 2609.31452v1…