PaperlayerView exact revision on arXiv

Source-linked AI summary

DRHeC: Differentiable Rendering for Hand-Eye Calibration with RGB-Based Gradients

Xiaotian Zhang, Yusheng Wang, Naoya Kagawa, Noritaka Takamura, Keiji Okuhara, Hiroyasu Baba, Jun Ota

arXiv:2609.36779v1cs.ROcs.CVcs.GR

TL;DR

Binary-mask differentiable rendering can lose internal profile details and provide unstable optimization guidance for hand-eye calibration. DRHeC combines RGB appearance cues with mask geometry and uses geometry-preserving real-to-simulation translation. Under the tested setup, it substantially improves over EasyHeC and achieves strong accuracy and robustness in simulation and real-world scenarios.

  • Problem

    Binary-mask differentiable rendering can lose internal profile details, causing reduced precision, unstable optimization, and possible pose ambiguity.

    Source: §I · p. 2, B. Differentiable Renderer Hand-eye… · p. 3

  • Method

    DRHeC combines RGB derivatives with mask centroid and area features, and uses mask-guided image-to-image translation to preserve geometric information.

    Source: §I · p. 2, A. I2IT · p. 5

  • Results

    DRHeC substantially improves over EasyHeC under the present setup and achieves higher success rates than several closed-form marker-based baselines under the tested conditions.

    Source: F. Discussion · p. 12

  • Takeaways & Limitations

    Experiments demonstrate strong accuracy and robustness in simulation and real-world scenarios, including strong calibration performance with the minimally color-varied DENSO VS060.

    Source: F. Discussion · p. 13, §VIII · p. 15

    The study does not provide a formal observability analysis or quantitative tolerance boundary for degraded single-frame observability.

    Source: F. Discussion · p. 13

Abstract

from arXiv · show

Accurate hand-eye calibration is crucial for precision manipulation. Traditional methods rely on markers, with their precision dependent on marker accuracy and observability. In contrast, markerless methods, such as learning-based approaches, use deep neural networks to directly extract keypoints or features from images, enabling the computation of hand-eye transformation with a single image and without the need for physical markers. Recently, differentiable rendering-based methods for hand-eye calibration have leveraged physical models to render binary masks and compare them with observations, enabling hand-eye calibration without fiducial markers in the calibration stage and providing interpretable optimization. While the state-of-the-art differentiable rendering methods achieve remarkable accuracy, the use of binary masks can result in the loss of internal profile details, reducing precision. Additionally, these methods can also suffer from unstable optimization and local minima. In this study, we propose a novel RGB-based differentiable rendering framework that provides richer geometric and appearance cues by incorporating color and mask geometric features, thereby improving calibration accuracy and optimization stability. Additionally, we propose a mask-guided image-to-image translation method to ensure explicit preservation of color and geometric consistency throughout the translation. Our approach is validated through both simulation and real-world experiments, with results demonstrating strong accuracy and robustness and clear improvements over existing differentiable rendering methods. Our method achieves a grasping success rate of 88.9% and insertion success rate of 57.4% on the UR5e real-world experiment, outperforming the state-of-the-art differentiable rendering hand-eye calibration method EasyHeC by 46.3 and 48.1 percentage points, respectively.

I. INTRODUCTION

Hand-eye calibration aligns camera and robot coordinates for vision-guided control, but existing markerless approaches face data, gradient, and mask-information challenges. DRHeC combines RGB derivatives with mask geometry, alongside mask-guided translation, to improve optimization and geometric consistency.

A. Hand-Eye Calibration

  • Hand-eye calibration estimates the end-effector-to-camera transformation, aligning camera coordinates with the robot for accurate visual processing and control.Marker-based methods depend on marker precision and measurement quality, while markerless learning-based approaches often depend on dataset scale and quality and can be susceptible to noise.Source: §I · p. 1

I. INTRODUCTION

  • Binary-mask rendering can leave calibration under-constrained because interior pixels provide no useful gradients, creating vanishing-gradient regions and possible pose ambiguity.Differentiable-rendering methods can also suffer from vanishing or exploding gradients and inadequate guidance toward the target pose.Source: §I · pp. 1–2
  • DRHeC combines RGB derivatives with mask geometry to improve calibration accuracy and optimization stability, and uses mask-guided I2IT to preserve geometric consistency under mild appearance variations.Its loss incorporates mask centroids, areas, and RGB derivatives; the I2IT method targets real-to-simulation alignment under mild illumination, color, and material variations.Source: §I · p. 2
  • The study collects two real-robot datasets of approximately 1,500 paired images each from UR5e and DENSO VS060 and reports real-world task improvements over binary-mask-based differentiable-rendering methods.Marker-based comparisons are provided as reference baselines under the tested setup.Source: §I · p. 2

B. Differentiable Renderer Hand-eye Calibration

Differentiable-rendering calibration can lose internal profile details with binary masks and may converge poorly. DRHeC adds RGB derivatives and mask geometry, while its I2IT component uses mask loss to reduce geometric inconsistencies in real-to-simulation translation.

B. Differentiable Renderer Hand-eye Calibration

  • Differentiable rendering computes gradients between rendered images and scene parameters, enabling iterative pose optimization by aligning rendered and real images.Prior robotics work includes distance-map and appearance-based pose estimation and EasyHeC’s combination of differentiable rendering with space exploration.Source: B. Differentiable Renderer Hand-eye… · pp. 2–3
  • DRHeC uses RGB derivatives and mask centroid and area features to address binary-mask detail loss, pose ambiguity, and inaccurate gradients in differentiable-rendering calibration.The comparison on synthetic UR5e shows EasyHeC and IPE struggling to converge from significant initial pose deviations, while DRHeC converges.Source: B. Differentiable Renderer Hand-eye… · p. 3, Figure 1 · p. 3

C. Image-to-Image Translation

  • CycleGAN-based real-to-simulation I2IT adds a mask loss to reduce geometric inconsistencies when robot model size discrepancies complicate translation.The section notes that paired datasets for conditional image translation can be challenging to acquire.Source: C. Image-to-Image Translation · p. 3

III. PRELIMINARIES

Differentiable rendering uses image gradients to optimize scene parameters against observations. In single-image hand-eye calibration, the method optimizes the hand-eye transform by aligning rendered and observed robot masks, but binary-mask supervision can be unstable and ambiguous.

A. Differentiable Rendering

  • Inverse rendering estimates scene parameters by minimizing pixel-wise squared error between a rendered image and the observation.Differentiable rendering supplies gradients from the 2D image to scene parameters for optimization.Source: A. Differentiable Rendering · p. 4

B. Differentiable Renderer Hand-Eye Calibration

  • Single-image calibration renders the known robot as a binary image from its joint configuration and candidate hand-eye transform, then optimizes the transform toward the observed mask.The robot base pose relative to the camera is initialized approximately; the joint configuration is known and is not optimized.Source: B. Differentiable Renderer Hand-Eye… · p. 4
  • The rendered observations must remain meaningfully aligned with sim-like images, so the robot geometry must be sufficiently accurate.Exact appearance consistency is not required because I2IT reduces the real-to-simulation appearance gap.Source: B. Differentiable Renderer Hand-Eye… · p. 4
  • Binary-mask calibration can be unstable and ambiguous because it lacks global gradient direction and color information.Optimization may have vanishing or exploding gradients, move the rendered mask out of frame, or yield pose ambiguity.Source: B. Differentiable Renderer Hand-Eye… · p. 4

IV. METHOD

The method combines RGB appearance cues with global mask geometry to guide hand-eye calibration. Its robot-specific I2IT translation aligns real images with the simulation domain while preserving geometric consistency.

IV. METHOD

  • RGB appearance cues plus centroid and area constraints provide local and global guidance intended to improve calibration precision and stabilize optimization.The geometric losses are used when initial pose error is large; RGB derivatives reduce the risk of local minima compared with binary-mask supervision.Source: §IV · pp. 4–5

A. I2IT

  • I2IT transforms real robot images into simulation-like images, and a separate model must be trained for each robot application.The translated and rendered images are compared, particularly when they share the same hand-eye transform and robot configuration.Source: A. I2IT · p. 5
  • The CycleGAN training objective includes identity, adversarial, and cycle-consistency losses, while a mask loss penalizes differences between generated and input binary masks.The mask loss is weighted in the objective, and the resulting translation is reported to improve geometric and color consistency.Source: A. I2IT · p. 5
  • Preliminary binary-image calibration is used only for coarse alignment of I2IT training pairs; the trained I2IT network remains fixed during final calibration.The preliminary estimate does not initialize or constrain final DRHeC optimization.Source: A. I2IT · p. 5
  • The calibration loop minimizes RGB MSE and adds centroid and area losses when centroid error exceeds a threshold, then freezes the rotation gradient.The rendered image and observed image provide the RGB loss; the threshold condition controls the geometric-loss and rotation-gradient steps.Source: A. I2IT · p. 5

B. RGB-Based Differentiable Rendering

The section replaces binary-mask-only supervision with RGB-based gradients and mask geometry to improve calibration precision and optimization stability. It combines smooth centroid and area losses with staged optimization to address uneven gradients and difficult early alignment.

B. RGB-Based Differentiable Rendering

  • RGB derivatives alone produced unsatisfactory results because their uneven gradient propagation can cause divergence and increase the likelihood of estimates going out of frame.The combined objective adds centroid-distance and mask-area losses to the original RGB loss.Source: B. RGB-Based Differentiable Rendering · pp. 5–6
  • Soft-weighted centroids provide smooth, differentiable image-position measurements that improve gradient flow compared with hard-edged binary masks.The centroid loss measures Euclidean distance between rendered and simulation-like image centroids.Source: B. RGB-Based Differentiable Rendering · p. 6
  • During coarse alignment, rotation gradients are frozen until centroid loss crosses a threshold, after which full RGB-loss optimization refines rotation.The threshold trades early-stage stability against the speed of rotation refinement.Source: B. RGB-Based Differentiable Rendering · p. 6
  • Mask-area loss supplements centroid alignment by stabilizing z-translation when aligned mask centers do not match the object’s true center.The area loss uses a smooth, differentiable approximation of object area from grayscale intensity.Source: B. RGB-Based Differentiable Rendering · p. 6

C. Implementation Details

The implementation uses CycleGAN settings for image translation and Adam for differentiable-rendering optimization, with fixed loss weights and a switching threshold. The authors note that weights and threshold may need retuning for substantially different environments or datasets.

C. Implementation Details

  • The differentiable-rendering optimization uses Adam at learning rate 0.002 for up to 2000 iterations, with RGB, centroid, and area loss weights of 1.0, 0.01, and 0.0001, respectively.The two-step optimization threshold is 0.25.Source: C. Implementation Details · p. 7
  • I2IT training follows original CycleGAN settings, sets λmask to 1.0, and runs for 400 epochs with a learning-rate schedule that decays after epoch 200.The λid, λGAN, and λcycle weights remain unchanged from the original settings.Source: C. Implementation Details · pp. 6–7
  • Loss weights and the switching threshold may require tuning for substantially different environments or datasets.The authors caution that imbalanced settings could cause instability or suboptimal convergence.Source: C. Implementation Details · p. 7

V. SIMULATIONS

The simulations evaluate the proposed method on synthetic and real-world datasets, using synthetic data to compare rendering-based methods under varying initial pose errors and real-world data to compare with learning-based methods.

V. SIMULATIONS

  • Synthetic experiments compare rendering-based methods under varying initial pose errors, while real-world experiments compare against learning-based hand-eye calibration methods.The study uses both synthetic and real-world datasets to evaluate the proposed method.Source: §V · p. 7

A. Synthetic Datasets

On the synthetic UR5e dataset, DRHeC has the lowest Q3 and mean translation and rotation errors across all tested perturbation levels. Adding mask-area loss further improves success rates, including under high perturbation.

A. Synthetic Datasets

  • The synthetic UR5e evaluation uses 200 test samples per perturbation level, with translation errors increasing across low, moderate, and high settings while rotation errors remain within ±π/18 radians.The methods compared are EasyHeC, IPE, RGB loss, RGB plus centroid loss (RGBC), and DRHeC.Source: A. Synthetic Datasets · pp. 7–8
  • DRHeC achieves the lowest Q3 and mean translation and rotation errors across all perturbation levels, with particularly strong performance under moderate and high perturbations.Fig. 3 shows error distributions for five methods across low, moderate, and high initial perturbations.Source: Figure 3 · p. 7, A. Synthetic Datasets · p. 8
  • Success rates under high perturbation rise from 97.5% with RGBC to 99.5% with DRHeC after adding mask-area gradients.The reported results associate the area-loss addition with further improvements in error metrics and success rates.Source: A. Synthetic Datasets · p. 8

B. Real World Datasets

The real-world evaluation tests hand-eye calibration on a Baxter dataset and UR5e manipulation tasks, with additional experiments spanning robot platforms and visual conditions. Across pixel thresholds, DRHeC outperforms learning-based and differentiable-rendering methods on the Baxter dataset.

B. Real World Datasets

  • The Baxter evaluation uses images from 20 robot poses and ground-truth keypoints; offline coarse alignment prepares I2IT training pairs but is not part of final DRHeC optimization.Together with UR5e and DENSO VS060 experiments, the results provide initial, non-exhaustive evidence across robot morphologies, appearances, and data sources.Source: B. Real World Datasets · p. 8
  • DRHeC outperforms learning-based and differentiable-rendering methods across 2D PCK thresholds from 20px to 200px on the real-world dataset.The dataset contains 100 Baxter images from 20 robot poses, annotated with ground-truth keypoints.Source: B. Real World Datasets · p. 8

VI. REAL-WORLD EXPERIMENTS

  • Real-world experiments evaluate DRHeC on UR5e grasping and insertion tasks and additionally test a nearly colorless DENSO VS060 robot.Grasping and insertion assess object pickup and accurate hole insertion after calibration.Source: §VI · p. 8, Figure 4 · p. 9

A. Experiment Setup

  • The UR5e setup assumes a calibrated camera and robot, uses SAM masks with manual correction when needed, and maintains stable indoor lighting.Mask preparation is outside the calibration optimization and can be replaced by another task-specific automatic method.Source: A. Experiment Setup · p. 9
  • AprilTags and PnP estimate object pose only for downstream task evaluation; they are not used in the hand-eye calibration procedure.The experiments compare DRHeC with marker-based methods and EasyHeC on grasping and insertion.Source: A. Experiment Setup · p. 9

B. I2IT Preprocessing

DRHeC’s I2IT preprocessing translates real images toward simulation style while preserving geometry, and its UR5e grasping results substantially exceed EasyHeC. Single3D (iterations) nevertheless has the highest grasping success rate in the tested setup.

B. I2IT Preprocessing

  • The I2IT training pairs comprise 1,470 images from 10 robot configurations, with preliminary mask-based calibration used only to align pairs during dataset preparation.That preprocessing calibration is not used in subsequent DRHeC optimization.Source: B. I2IT Preprocessing · p. 9
  • The translated images preserve geometric information while changing color correspondences, with only minor pose discrepancies attributed to mask extraction and model differences.All 10 robot configurations were successfully translated to the simulation-image domain.Source: B. I2IT Preprocessing · p. 9
  • DRHeC achieves the lowest MSE and MAE, highest PSNR, and higher mask IoU than other I2IT approaches, although CycleGAN has a slightly higher SSIM score.DRHeC’s SSIM is 0.9722 versus CycleGAN’s 0.9725; visual comparisons also report more consistent color translation in challenging poses and joint regions.Source: B. I2IT Preprocessing · pp. 9–10, Table III · p. 9, Figure 6 · p. 10

C. Grasping Experiments

  • Single3D (iterations) achieves the highest grasping success rate, while DRHeC improves substantially over EasyHeC and performs strongly against several closed-form marker-based baselines.The marker-based success rates may be affected by small marker image regions at the approximately 2 m camera distance, pose selection, and marker precision.Source: C. Grasping Experiments · p. 10
  • 88.9% versus EasyHeC’s 42.6%: DRHeC achieves a higher grasping success rate without fiducial markers for hand-eye calibration.Single3D (iterations) remains the strongest overall method in these grasping trials and relies on fiducial-marker observations.Source: C. Grasping Experiments · pp. 10–11, Table IV · p. 11

D. Insertion Experiments

On UR5e insertion, DRHeC substantially exceeds EasyHeC but remains below Single3D (iterations); the task applies a stricter collision-as-failure criterion. DENSO VS060 experiments also report strong DRHeC results despite the robot’s nearly uniform appearance.

D. Insertion Experiments

  • 57.4% versus EasyHeC’s 9.3%: DRHeC achieves a higher insertion success rate, although Single3D (iterations) remains highest.Insertion is more difficult than grasping, and any collision with the object counts as failure.Source: D. Insertion Experiments · p. 11, Table V · p. 12
  • EasyHeC and Daniilidis both achieve 9.3% insertion success, but EasyHeC deviates only slightly from the target while marker-based methods tend toward larger errors, especially in translation.The comparison is reported for the tested setup.Source: D. Insertion Experiments · p. 11
  • For a representative insertion trial, DRHeC’s translation and rotation errors are 0.03 m and 0.02 rad, respectively, relative to Single3D (iterations) as a reference, not ground truth.EasyHeC’s corresponding errors are 0.05 m and 0.03 rad; DRHeC is closest to the reference in this trial.Source: D. Insertion Experiments · p. 11

E. DENSO VS060 Experiments

  • On DENSO VS060, DRHeC exceeds EasyHeC by 22.2 and 44.5 percentage points in grasping and insertion, respectively, while Single3D (iterations) has the highest success rates overall.The robot has nearly uniform visual appearance, making it a challenging test of RGB-based optimization.Source: E. DENSO VS060 Experiments · p. 11, Table V · p. 12
  • In a representative DENSO VS060 trial, DRHeC is closest to the Single3D (iterations) reference across translation and rotation, though that reference is not absolute ground truth.The reported translation and rotation errors for DRHeC are 0.03 m and 0.03 rad.Source: E. DENSO VS060 Experiments · p. 12

F. Discussion

DRHeC outperforms EasyHeC and several closed-form marker-based baselines in the tested setup, while its discussion identifies model, visibility, and optimization limits. Ablations support combining RGB, mask geometry, and image translation for robust calibration.

F. Discussion

  • DRHeC substantially improves over EasyHeC and achieves higher grasping and insertion success rates than several closed-form marker-based baselines under the tested setup.The authors note that the 2-meter calibration distance, 10 camera poses, and limited precision of printed markers may have affected marker-based results.Source: F. Discussion · p. 12
  • The method remains constrained by single-image local minima, imperfect real-to-simulation alignment, indoor-lighting assumptions, and the absence of formal observability limits.Extreme illumination or robot-mask occlusion may reduce reliability; deliberate appearance changes and broader platform generalization also lack systematic evaluation.Source: F. Discussion · pp. 12–13
  • Binary-mask ambiguity is presented as a qualitative case: different poses can yield nearly identical masks, while RGB cues may help distinguish them.The paper does not quantitatively evaluate ambiguity-specific cases.Source: F. Discussion · p. 13, Figure 8 · p. 13
  • DRHeC performs strongly on the minimally color-varied DENSO VS060, suggesting mask centroid and area features provide useful global information when color cues are limited.Source: F. Discussion · p. 13
  • At 1% geometric mismatch, mean translation error rises from 7.41 to 21.92 mm, while rotation error changes from 0.0197 to 0.0190 rad.The simulation tested 0%, 1%, 3%, and 5% globally anisotropic Gaussian scaling perturbations over 20 paired UR5e trials.Source: F. Discussion · p. 13
  • DRHeC takes about 14 seconds per calibration at 2,000 iterations, comparable to EasyHeC and considered practical for offline or occasional recalibration.Further acceleration would benefit online recalibration or high-throughput deployment.Source: F. Discussion · pp. 13–14

VII. ABLATION STUDY

  • Across loss-term ablations, full DRHeC performs best, followed by RGB plus centroid and RGB alone; the selected weights are λcentroid = 0.01 and λarea = 0.0001, with threshold = 0.25.The selected configuration was used across simulation and real-world experiments without environment-specific retuning.Source: §VII · p. 14, Table VI · p. 14, Table VII · p. 14
  • With no real-world hand-eye ground truth, I2IT is assessed through image metrics, which improve when translated images are used.The authors report lower MSE and MAE and higher PSNR and SSIM, interpreting this as a reduced real-to-synthetic appearance gap.Source: §VII · pp. 14–15, Table IX · p. 15

VIII. CONCLUSION

The conclusion presents a differentiable-rendering hand-eye calibration framework combining RGB derivatives and mask geometry, alongside mask-guided image translation. Its current implementation is limited by model inaccuracies, eye-to-hand configuration, and offline or setup-time operating conditions.

VIII. CONCLUSION

  • The framework combines RGB derivatives and mask geometry in differentiable rendering, while its modified I2IT loss preserves essential geometric information.Source: §VIII · p. 15
  • Reliance on a robot model can introduce calibration errors, and the current implementation is limited to eye-to-hand configuration because it often requires capturing the entire robot.Partial robot observations are proposed as a future route toward eye-in-hand calibration.Source: §VIII · p. 15
  • The current pipeline is mainly suited to offline or setup-time calibration, indoor scenes with moderate lighting, and unobstructed robot visibility.Broader validation, formal observability analysis, independent uncertainty evaluation, and wider model-deviation sensitivity studies remain future work.Source: §VIII · p. 15
Loading 2609.36779v1…