PaperlayerView exact revision on arXiv

Source-linked AI summary

AURORA: Active Uncertainty-Driven Re-Orientation for In-Hand Reconstruction

Feiyu Zhao, Yuetong Li, Chenxi Xiao

arXiv:2609.08493v1cs.ROcs.CV

TL;DR

Severe hand–object occlusion and passive reorientation make it difficult to obtain informative views for reconstructing grasped objects. AURORA uses uncertainty-guided planning to close the loop between reconstruction and in-hand rotation. It improves reconstruction and information-acquisition efficiency over non-active rotation strategies, while Ray-GPIS surpasses adapted active view-planning baselines on reported measures.

  • Problem

    Passive or open-loop manipulation does not explicitly select informative viewpoints, leaving occluded regions potentially under-observed and exploration time-inefficient.

    Source: §1 · p. 2

  • Method

    AURORA couples online object-centric reconstruction with in-hand reorientation, using Ray-GPIS direction-wise uncertainty to plan actions and fusing RGB-D observations in an object frame.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    AURORA outperforms non-active rotation strategies in reconstruction quality and information-acquisition efficiency, while Ray-GPIS surpasses adapted active view-planning baselines in reconstruction performance, action-ranking quality, and planning efficiency.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The framework progressively exposes under-observed regions and improves reconstruction completeness under severe hand–object occlusion.

    Source: §6 · p. 9

    Reconstruction remains sensitive to 6D pose-tracking errors under weak visual texture or severe hand–object occlusion.

    Source: §5 · p. 9

Abstract

from arXiv · show

Observing objects grasped by a robot hand is challenging due to severe visual occlusions. Although in-hand manipulation can expose hidden surfaces, existing approaches often rely on predefined or open-loop reorientation strategies that do not explicitly target under-observed regions. We propose AURORA, an active 3D reconstruction framework that closes the loop between online object-centric reconstruction and in-hand reorientation. At its core, Ray-GPIS estimates direction-wise reconstruction uncertainty along candidate viewing rays and selects next-best-view targets using an uncertainty--novelty objective, which are realized through an axis-conditioned in-hand rotation policy. The resulting RGB-D observations are fused incrementally using CAD-free 6D pose tracking and lightweight geometric reconstruction. Experiments demonstrate that AURORA improves reconstruction quality and information-acquisition efficiency over non-active rotation strategies, while Ray-GPIS also outperforms active view-planning baselines in reconstruction performance, action-ranking quality, and planning efficiency. Targeted ablations further validate its robustness to hand occlusion and pose errors. The project webpage is available at https://aurorahand.github.io/

1 Introduction

Hand–object occlusion and limited camera viewpoints leave object geometry partially observed, while passive or open-loop manipulation does not choose where to look next. AURORA closes the loop between reconstruction and reorientation, using uncertainty to guide actions and reporting improved completeness and exploration efficiency.

1 Introduction

  • Passive manipulation sequences do not reason about viewpoint informativeness or turn perceptual uncertainty into actions, so occluded regions may remain under-observed.The paper identifies this planning gap as a source of time-inefficiency.Source: §1 · p. 2
  • AURORA couples online object-centric reconstruction with in-hand reorientation, selecting actions to improve coverage of under-observed regions.It fuses the collected views for autonomous reconstruction through coordinated hand–eye interactions.Source: §1 · p. 2
  • Ray-GPIS converts GPIS point-wise uncertainty into direction-wise view scores for active in-hand reorientation.This is the paper's named planning contribution.Source: §1 · p. 2
  • The proposed framework and online fusion pipeline are reported to improve reconstruction completeness and exploration efficiency under hand–object occlusion.The introduction describes the evaluations as extensive and real-world.Source: §1 · p. 2

2 Related Works

Prior work studies in-hand manipulation, object reconstruction under occlusion, and active exploration, but does not address the coupled challenges of active continuous in-hand reconstruction. The paper positions AURORA against these lines of work and their limits.

2.1 In-Hand Manipulation for Object Reconstruction

  • Prior in-hand methods often prioritize stability, speed, or task success; NeuralFeels reconstructs from given visuotactile sequences but lacks active planning, and hardware differences preclude direct benchmarking.The paper notes limited comparisons with NeuralFeels in Appendix F.7.Source: §2.1 · pp. 2–3

2.2 Object Reconstruction and Tracking under Occlusion

  • Image-sequence reconstruction methods generally assume static geometry, while in-hand reconstruction faces unpredictable object motion and hand occlusion; related trackers address pose challenges but not active viewpoint selection.The cited related works primarily reconstruct from observed sequences rather than selecting actions to reveal unseen regions.Source: §2.2 · p. 3

2.3 Active Exploration and Uncertainty-Driven View Planning

  • Active exploration includes mobile sensing and object reposing, but prior approaches largely assume controlled or external manipulation rather than continuous in-hand action under persistent hand occlusion.These settings do not address action selection for revealing uncertain regions while the object moves in the hand.Source: §2.3 · p. 3

3 Methodology

AURORA closes the loop between object-centric reconstruction and in-hand reorientation, using reconstruction uncertainty to guide informative actions. It incrementally fuses tracked RGB-D observations and produces a watertight final mesh.

3 Methodology

  • AURORA iterates through in-hand rotation, object tracking and keyframe selection, point-cloud fusion, and uncertainty-based next-best-view planning before extracting a watertight mesh.The loop starts from an initial observation and repeats until termination.Source: §3 · p. 4

3.1 In-Hand Object Reorientation

  • The action space comprises three hand-frame rotation-axis primitives, and a tactile-based policy executes the selected axis on the grasped object.The feasible axes are −x, −y, and +z.Source: §3.1 · p. 4

3.2 Ray-GPIS Active View Planning

  • Ray-GPIS converts point-wise GPIS uncertainty into direction-wise planning scores by casting object-centered rays and aggregating posterior variance near predicted surface locations.The GPIS model is fitted to observed hit-ray anchors; miss-ray anchors serve only as query locations.Source: §3.2 · p. 4, §3.2.1 · p. 5

3.2.1 Ray-GPIS Uncertainty Estimation

  • Ray-GPIS samples directions from the point-cloud bounding-box center and uses normalized radial coordinates to probe around and beyond the observed extent.For miss rays, anchor depth is interpolated from nearby hit-ray depths, enabling uncertainty queries in unobserved directions.Source: §3.2.1 · pp. 4–5

3.2.2 Next-Best-View Selection

  • Next-best-view selection multiplies direction-wise uncertainty by a novelty weight based on distance from the current point cloud, favoring uncertain regions far from fused observations.Novelty is normalized by the median nearest-neighbor spacing, representing typical point-cloud resolution.Source: §3.2.2 · p. 5

3.2.3 Action Selection

  • Action selection transforms the chosen object-frame viewing direction into the hand-centered frame and selects the discrete action axis most aligned with the desired rotation.The desired rotation aligns that direction with the camera direction expressed in the hand-centered frame.Source: §3.2.3 · p. 5

3.3 Object-Centric Tracking and Reconstruction

  • CAD-free RGB-D pose tracking and visibility-aware keyframe filtering support incremental fusion of masked depth into a compact object-centric point cloud for planning.The final fused point cloud is converted to a watertight mesh using globally consistent normals and NKSR reconstruction.Source: §3.3 · p. 6

4 Experiments

AURORA is evaluated on six graspable real-world objects, with reconstruction quality and planning compared under fixed interaction budgets. It outperforms non-active rotation strategies and adapted active view-planning baselines, while targeted ablations examine robustness to occlusion and pose errors.

4.1 System Setup and Task Protocol

  • The real-world protocol uses a fixed RGB-D camera and Leap Hand, with 6 s rotations and replanning under a fixed 30 s interaction budget.Online point clouds and final offline meshes are evaluated against ground-truth meshes captured with a 3D scanner.Source: §4.1 · p. 6

4.2 Reconstruction Accuracy

  • After a fixed 30 s budget, offline mesh refinement raises mesh-level F@5 to 0.8624 while preserving F@10 of 0.9672; AURORA also outperforms non-active rotation baselines.The online point-cloud reconstruction reaches average F@10 of 0.9671 and F@5 of 0.8249.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3.1 Comparison with Non-active Baselines

  • Under the 30 s budget, AURORA's F@5 improves faster than open-loop schedules, while single-axis rotations improve more slowly and retain higher uncertainty.The predefined multi-axis schedule performs better than single-axis rotations but remains suboptimal because it cannot adapt to the current reconstruction state.Source: §4.3.1 · p. 8, Figure 5 · p. 8

4.3.2 Comparison with Active View-Planning Baselines

  • Ray-GPIS achieves the highest F-AUC and final F@5, obtains substantially higher Corr. than both adapted baselines, and requires substantially less planning time than ActNeRF.The planner comparison covers 120 paired simulation episodes.Source: §4.3.2 · p. 9, Table 2 · p. 9

4.4 Ablation and Robustness Analysis

  • Removing miss-ray interpolation reduces Unseen R-AUC under finger occlusion, while removing receptive-field integration causes a larger Corr. degradation under pose perturbations.The ablations associate miss-ray interpolation with reasoning about unobserved regions and receptive-field integration with less sensitive action ranking under small geometric misalignments.Source: §4.4 · p. 9, Table 2 · p. 9

5 Limitations

The framework has limitations in pose tracking, replanning timing, and manipulation stability.

5 Limitations

  • Pose tracking remains sensitive to errors under weak visual texture or severe hand–object occlusion.The authors identify tactile or contact-aided tracking as a motivation for future work.Source: §5 · p. 9
  • The fixed 6 s replanning interval could instead adapt to uncertainty or tracking confidence..Source: §5 · p. 9
  • Occasional object slip or drops indicate a need for more robust control and sim-to-real transfer..Source: §5 · p. 9

6 Conclusion

AURORA couples online reconstruction with uncertainty-driven in-hand reorientation, using Ray-GPIS to guide rotations and progressively expose under-observed regions. Its evaluations show improvements over non-active and active-planning baselines, while the appendix describes the hardware, policy deployment, and tactile sensing setup.

6 Conclusion

  • AURORA closes the loop between online object-centric reconstruction and uncertainty-driven reorientation; experiments report better reconstruction quality and information-acquisition efficiency than non-active strategies.Ray-GPIS also surpasses adapted active view-planning baselines in reconstruction performance, action-ranking quality, and planning efficiency.Source: §6 · p. 9

A.1 Hardware Setup

  • The setup uses an Azure Kinect DK RGB-D camera and Leap Hand, with an EinScan Pro 2X V2 scanner for ground-truth meshes.Real-world deployment runs on a desktop with an Intel Core i7-13700 CPU, 32 GB RAM, and an NVIDIA RTX 4090D GPU.Source: §A.1 · p. 13

A.2 Policy Deployment on the Leap Hand

  • The Leap Hand policy takes an axis command and outputs joint-level motor commands for rotating the grasped object about that axis.The authors retrain the policy in simulation for the Leap Hand morphology and tactile sensing setup.Source: §A.2 · p. 13

A.3 FSR Sensor Layout and Usage

  • FSR sensors across the fingertips, phalanges, and palm provide tactile feedback to the low-level reorientation policy, not the reconstruction pipeline.Their benefit to reconstruction is indirect, through more reliable object reorientation.Source: §A.3 · p. 13

A.4 FSR Sensor Activation and Utilization

The appendix describes how tactile feedback, Ray-GPIS uncertainty, next-view mapping, pose tracking, and point-cloud fusion support AURORA’s online reconstruction. It also reports planner runtimes and explains the replanning interval.

A.4 FSR Sensor Activation and Utilization

  • Real-hardware FSR activations are sparser than simulated activations, but consistent feedback from key contact regions still supports successful manipulation.The passage attributes the discrepancy to sensor force thresholds, hysteresis, and object stiffness differences, and suggests sensor placement, calibration, and signal normalization as possible improvements.Source: Figure A.2 · p. 14, §A.4 · p. 15

B.1 Hit/Miss Ray Assignment and Anchor Interpolation

  • For miss rays, Ray-GPIS interpolates anchor depths from nearby hit rays, and excludes a ray from the planning update when no valid neighbor exists.This lets it evaluate uncertainty near plausible surface locations even without direct observations.Source: §B.1 · pp. 15–16

B.2 GPIS Posterior Variance

  • Ray-GPIS uses GP posterior variance as geometric uncertainty, with high variance in less-observed regions and low variance near well-observed anchors.The GP model includes observation noise variance.Source: §B.2 · p. 16

C Details of NBV-to-Action Mapping

  • The NBV direction is transformed into the hand-centered world frame, converted to a minimal alignment rotation, and projected onto the low-level policy’s discrete action set.The selected primitive has an axis most aligned with the desired viewpoint-correcting rotation.Source: C Details of NBV-to-Action Mapping · pp. 16–17

D Details of 6D Pose Tracking and Keyframe Selection

  • BundleTrack estimates object pose from segmented RGB-D observations, while a keyframe filter retains frames meeting reliability and informativeness criteria for point-cloud fusion.The filter addresses unreliable frames caused by hand–object occlusion and imperfect segmentation, and seeks sufficient viewpoint change without redundant frames.Source: D Details of 6D Pose Tracking and… · pp. 17–18

E Details of Mesh Extraction; F.1 Planner and Reconstruction Hyperparameters

  • AURORA plans online from a lightweight point cloud; watertight mesh extraction is optional post-processing for evaluation and visualization, and Table A.1 lists planner and reconstruction hyperparameters.The resulting mesh is not used by the online planner, which avoids repeated mesh-extraction latency.Source: E Details of Mesh Extraction · p. 18, §F.1 · p. 18, Table A.1 · p. 18

F.2 Runtime and Replanning Interval

  • 261 ms, 0.072 ms, and 2.42 ms per update are the reported times for Ray-GPIS GP update, candidate scoring, and NBV-to-action mapping, respectively.Segmentation and BundleTrack require approximately 256.9 ms per RGB-D frame; the low-level controller’s roughly 6 s rotation and stabilization time determines the replanning interval.Source: §F.2 · p. 19

F.3 Robustness to 6D Pose-Tracking Errors

AURORA retains most of its unperturbed F@5 performance under the tested pose-tracking perturbations.

F.3 Robustness to 6D Pose-Tracking Errors

  • 96.3% and 90.3% of unperturbed F@5 are retained under rotational/translational pose perturbations of 5°/5 mm and 10°/10 mm, respectively.The passage reports gradual degradation rather than immediate pipeline failure under these moderate pose errors.Source: §F.3 · p. 19

F.4 Additional Comparison with Open-loop Rotation Baselines

AURORA replans from the current reconstruction state, maintaining faster F-score improvement and reaching higher accuracy sooner than open-loop rotation baselines. Fixed-axis and scheduled rotations can leave regions occluded or fail to adapt to object slip.

F.4 Additional Comparison with Open-loop Rotation Baselines

  • Single-axis rotations leave some surfaces persistently occluded, while the predefined Z-X-Y-Z-X schedule cannot target details based on the current reconstruction state.AURORA selects the next rotation axis using estimated reconstruction uncertainty to target under-observed regions.Source: §F.4 · p. 20
  • AURORA maintains faster F-score improvement and reaches higher accuracy sooner on average than open-loop baselines.The temporal Mesh–Mesh F-score curves average six objects at 2 mm, 5 mm, and 10 mm tolerances.Source: Figure A.4 · p. 20, §F.4 · p. 20
  • After approximately 18 s, F-score curves tend to saturate as surfaces become observed and sensor, tracking, fusion, and mesh-resolution limits constrain further gains.Newly recovered regions can be offset in fixed-threshold scores by distortions or misalignment elsewhere in the extracted mesh.Source: §F.4 · p. 20
  • Under sim-to-real differences in tactile feedback, contact dynamics, and friction, x- and y-axis rotations are less efficient and stable than z-axis rotation and more prone to slip or unstable motion.These hardware constraints can worsen single-axis baseline reconstruction under the same manipulation budget.Source: §F.4 · p. 20

F.5 Comparison with Single-View 3D Reconstruction Baselines

AURORA achieves higher mean Mesh–Mesh F-scores than TRELLIS.2 and SPAR3D at the 5 mm and 10 mm thresholds across evaluated objects. The comparison also describes distinct single-view reconstruction errors and evaluation constraints.

F.5 Comparison with Single-View 3D Reconstruction Baselines

  • AURORA achieves higher mean Mesh–Mesh F-scores than TRELLIS.2 [43] and SPAR3D [44] at 5 mm and 10 mm thresholds across all evaluated objects.The single-view baselines provide a reference from incomplete visual observations, although they are not designed for active in-hand reconstruction.Source: §F.5 · p. 21
  • TRELLIS.2 [43] often leaves open surfaces or over-smooths details, whereas SPAR3D [44] tends toward watertight but geometrically misaligned meshes.The cited examples include incomplete cutting planes and warped or non-orthogonal geometry.Source: §F.5 · p. 21, Figure A.5 · p. 21
  • The single-view baselines rely on clean high-resolution images, lack metric scale, and hallucinate appearance for occluded regions; AURORA incrementally fuses observations into metrically consistent reconstructions.The reported baseline meshes were rescaled to ground truth before Mesh–Mesh metric evaluation.Source: §F.5 · p. 22

F.6 Downstream Manipulation Evaluation

Full Ray-GPIS produced the highest reported success rates for stable placement, regrasp, and the combined task, testing whether reconstructed meshes support downstream manipulation.

F.6 Downstream Manipulation Evaluation

  • Evaluation compared single-view reconstruction, a non-active fixed rotation schedule, adapted active planners, and Full Ray-GPIS on stable placement followed by regrasp.Each method was evaluated in 30 trials per object across 10 objects.Source: §F.6 · p. 22
  • 57.7%, 77.7%, and 45.0% success for placement, regrasp, and complete placement-and-regrasp, respectively, were highest with Full Ray-GPIS.It consistently outperformed the fixed schedule, adapted active planners, and single-view reconstruction baselines.Source: §F.6 · p. 22

F.7 Comparison with NeuralFeels & Discussions

AURORA selects closed-loop reorientations from reconstruction uncertainty, unlike NeuralFeels’ fixed-axis open-loop visual-and-tactile sequence. A limited Pepper comparison reports a higher AURORA score and shorter reconstruction time, but differing setups and texture constrain interpretation.

F.7 Comparison with NeuralFeels & Discussions

  • NeuralFeels uses fixed-axis open-loop rotations with visual and tactile sequences, while AURORA selects closed-loop reorientations from reconstruction uncertainty without tactile sensing.The authors note that the differing task formulations make a controlled direct comparison difficult.Source: §F.7 · p. 22
  • AURORA scored F@5 0.89 versus NeuralFeels’ average score 0.76 on Pepper, but the authors caution this does not establish strict superiority.The comparison uses a matched 30 s interaction budget; hardware, sensing, policies, task focus, and object instances differ.Source: §F.7 · pp. 22–23, Table A.4 · p. 23
  • AURORA requires sufficient visual texture and may fail on weakly textured objects; richer texture may partly explain the observed score gap.The paper says the Pepper object had a similar shape but richer texture than the object shown in the NeuralFeels video.Source: §F.7 · p. 23
  • For the same 30 s observation sequence, AURORA’s pipeline took 71.82 s versus NeuralFeels’ 417.13 s, mainly using lightweight point-cloud fusion and planning instead of dense neural SDF optimization.The AURORA runtime includes mesh extraction.Source: §F.7 · p. 23
Loading 2609.08493v1…