PaperlayerView exact revision on arXiv

Source-linked AI summary

GLoTouch: Global-to-Local Haptic Perception Using a Parallel Gripper for Object Search, Recognition, and Grasping Without External Vision

Zonglin Li, Wanruo Zhang, Yiming Wang, Kun Song, Xinyi Zhou, Daolin Ma

arXiv:2609.27695v1cs.RO

TL;DR

When external vision is unreliable in dark or low-light environments, GLoTouch addresses the challenge of connecting container-scale search with fine-grained object recognition using a parallel gripper. It uses a probe for global haptic exploration and local tactile observations matched to a target 3-D model; reported end-to-end retrieval succeeds in simulation and on hardware.

  • Problem

    Dark or low-light environments can make external cameras unreliable for object localization and geometry, while touch-only retrieval must connect container-scale exploration with local recognition.

    Source: §I · p. 1

  • Method

    GLoTouch uses a passive long-reach probe for force-based global search and candidate geometry estimation, then uses the same gripper’s tactile sensors to match local observations to the target STL without object-specific training.

    Source: §I · p. 1

  • Results

    End-to-end retrieval in five-object scenes succeeds at 84.0% (84/100) in simulation and 76.0% (38/50) on hardware.

    Source: §VII · p. 8

  • Takeaways & Limitations

    The reported experiments show that global-to-local haptic coupling can retrieve a model-specified object from a multi-object container without external vision.

    Source: §VII · pp. 7–8

Abstract

from arXiv · show

Perceiving objects in the environment is a fundamental capability of autonomous robots. In dark or low-light environments, external cameras often fail to reliably perceive object positions and geometry; when visual sensing is unavailable, completing target search, recognition, and grasping through touch alone becomes a key robot manipulation capability. This task must simultaneously address container-scale spatial exploration and object-scale fine-grained geometric perception, which is particularly challenging for low-degree-of-freedom parallel grippers. However, a unified framework remains lacking for connecting container-scale spatial exploration with object-scale fine-grained geometric perception and grasping. To address this challenge, we present \textbf{GLoTouch}, a global-to-local haptic perception and manipulation framework built on a parallel gripper. In the global stage, the gripper holds a passive long-reach probe, combining force measurements with known tool geometry to localize contacts and actively estimate candidate-object positions, coarse contours, and heights. In the local stage, the robot sets down the probe and uses the bilateral visuotactile sensors on the same gripper to directly acquire local haptic observations, which are matched against a given target 3-D model without object-specific training. We evaluate the framework in both simulation and real-robot experiments. Source code will be open-sourced.

I. INTRODUCTION

GLoTouch addresses touch-only retrieval when external vision is unreliable by linking container-scale search with local recognition and grasping using one parallel gripper. Its probe-mediated global stage estimates candidate geometry, while a training-free matcher compares local tactile observations with the target model.

I. INTRODUCTION

  • Touch-only retrieval matters where dark or poorly illuminated scenes make camera-based object localization and geometry unreliable.The task is to search for, recognize, and retrieve the object matching a supplied 3-D model from a multi-object container.Source: §I · p. 1
  • Local contact is inherently limited, movable objects may shift under force, and global search and local recognition require different sensing capabilities.The global stage needs long reach and sensitive collision detection; local recognition needs stable contact and spatially resolved geometry.Source: §I · p. 1
  • Global probe-mediated force exploration estimates candidate positions, contours, and heights; local training-free STL matching identifies the target from tactile observations.The same gripper performs search, local recognition, and retrieval.Source: §I · p. 1
  • 84.0% in simulation and 76.0% on hardware are the reported end-to-end retrieval success rates in five-object containers.The paper reports these results as validating tactile-only target retrieval.Source: §I · p. 1

A. Tactile Object Search and Retrieval

  • Prior retrieval systems use dexterous hands and learned tactile policies, while GLoTouch combines a parallel gripper with a temporary probe and training-free STL matching.The probe extends contact sensing to coverage-driven container-scale search, and the gripper remains available for recognition and grasping.Source: A. Tactile Object Search and Retrieval · p. 2

B. Tool-Mediated Exploration and Extrinsic Contact Sensing

The paper frames touch-only retrieval in a dark or low-light container where object locations and geometries are unknown but a target STL model is given. GLoTouch uses a passive probe for global discovery and coarse estimation, then structured local tactile observations for model matching rather than complete reconstruction.

B. Tool-Mediated Exploration and Extrinsic Contact Sensing

  • A passive, uninstrumented spherical-tipped probe supports global discovery and coarse geometric estimation before the same gripper performs local tactile recognition and grasping.The probe is used for global exploration; the gripper then resumes its end-effector role.Source: B. Tool-Mediated Exploration and… · p. 3

C. Active Tactile Geometry Acquisition and Object Recognition

  • GLoTouch selects local actions from the globally estimated coarse contour and compares local tactile geometry and jaw support widths directly with the target model.It avoids aiming for complete reconstruction or fusing point clouds in a fixed world frame, which would add state-estimation and data-association problems amid unknown object motion.Source: C. Active Tactile Geometry Acquisition… · p. 3

A. Task Formulation

  • The task assumes no external camera observations during control and restricts sensing to proprioception, wrist force–torque, fingertip tactile imaging, and known model geometry.The target is the physical object matching the supplied STL mesh; object locations, poses, and geometries are initially unknown.Source: A. Task Formulation · p. 3

B. System Setup

The system uses a parallel gripper and wrist force–torque sensing to infer probe contacts during global exploration. After setting down the unsensored probe, the gripper uses bilateral tactile depth maps and jaw aperture for local geometric matching.

B. System Setup

  • The seven-degree-of-freedom Flexiv arm uses wrist force–torque sensing and an Xense parallel gripper; probe-tip contact is inferred from wrench measurements and robot kinematics.The probe has no embedded sensor and is placed at a designated holder after global exploration.Source: B. System Setup · p. 3
  • During the global stage, the system observes the TCP transform for contact detection and safety stopping.Source: B. System Setup · p. 3
  • The local stage uses left and right tactile depth maps plus parallel-jaw aperture as its geometric observation for model matching.Stable tactile depth and aperture measurements supply the local observation.Source: B. System Setup · p. 3

C. Global-to-Local Pipeline

GLoTouch moves from probe-mediated container-scale exploration to local candidate matching and grasping. Contact detection triggers guarded stops, while known probe geometry and reaction measurements support contact localization.

C. Global-to-Local Pipeline

  • The robot searches at container scale with a held probe, then sets it down to acquire local observations, rank candidates against the target model, and grasp the lowest-loss match.The pipeline uses the same gripper for probe-mediated exploration and direct local contact.Source: C. Global-to-Local Pipeline · p. 3

A. Contact Detection and Active Retreat

  • A stationary unloaded interval establishes the wrench baseline; filtered residuals drive soft contact thresholds, while separate absolute and residual limits trigger hard stops.After normal contact, the robot retains the onset pose and wrench, retreats, and resumes tracing only after wrench and velocity stabilize.Source: A. Contact Detection and Active Retreat · p. 4

B. Tool-Geometry-Aware Contact Localization

  • At contact, the method combines approach direction with opposing reaction and known probe geometry to infer the object-facing direction and support-plane intercept.An initial center hypothesis is placed inside the boundary and refined as contacts accumulate.Source: B. Tool-Geometry-Aware Contact… · p. 4

C. Coverage-Driven Object Search

Coverage-driven search tracks executed paths and samples frontier candidates within the permitted container boundary. It selects the nearest unexplored frontier and resumes searching after each object is characterized.

C. Coverage-Driven Object Search

  • The policy tracks executed-path samples and samples frontier candidates uniformly within the permitted container boundary.A candidate remains unexplored according to its distance from the executed-path samples and the coverage radius.Source: C. Coverage-Driven Object Search · p. 4
  • Traced object centers enter an exclusion list, and duplicate contacts within an object's envelope are skipped before global search resumes.A contact switches the search to local boundary tracing.Source: C. Coverage-Driven Object Search · p. 4
  • The search continues until all expected objects are found or the search space is exhausted.After characterization, the nearest unexplored frontier is selected to produce continuous Cartesian segments through the container.Source: C. Coverage-Driven Object Search · p. 4

D. Orbit-and-Probe Contour Estimation

Orbit-and-probe tracing gathers boundary contacts from known approach directions, fits a coarse contour, and estimates object height for the local stage.

D. Orbit-and-Probe Contour Estimation

  • The robot follows a clearance orbit and approaches radially, avoiding chord paths through the object while producing boundary observations with known approach directions.After each accepted contact, it retreats, updates the center, and prioritizes the largest uncovered angular interval.Source: D. Orbit-and-Probe Contour Estimation · p. 4
  • For round-like candidates, contact points are fit with a circle; persistently high circle residuals prompt additional contacts and an oriented-rectangle fit.The selected primitive supplies center, coarse contour class, scale, and, for box-like shapes, planar orientation.Source: D. Orbit-and-Probe Contour Estimation · p. 4
  • Object height is estimated from top-contact height minus calibrated floor height, and the candidate record carries center, height, scale, orientation, contour class, and local-fit quality.The probe is positioned above the fitted center and lowered using the guarded contact detector.Source: D. Orbit-and-Probe Contour Estimation · pp. 4–5

E. Contact-Mode Validation

Probe-mediated tracing fits the calibration-square contour better than bare-gripper tracing, whose larger contact uncertainty yields greater center error and scatter. The local stage filters tactile acquisitions and matches finite windows to a target model.

E. Contact-Mode Validation

  • Probe-mediated tracing fits the calibration-square contour well, while bare-gripper tracing has larger center error and contact scatter with a misaligned fit.The authors attribute the difference to uncertainty in the effective contact point caused by the gripper's large envelope.Source: E. Contact-Mode Validation · p. 5

A. Stable Tactile Acquisition and Quality Assessment

  • Force-limited closures at multiple height layers are accepted only when aperture plateaus and TCP motion is small; tactile depth maps are filtered into contact patches for quality assessment.Sequence quality uses median nonplanarity across valid left/right patches and the fraction of valid patches; bilateral layers receive the highest weight.Source: A. Stable Tactile Acquisition and… · p. 5

B. Target-Conditioned Finite-Window Matching

  • Global contour class determines whether each candidate is sampled at three independent yaws or along three parallel longitudinal tracks, producing three multi-height sequences.Reference TCP positions and rotations define an action graph whose coupling probability uses longitudinal span and relative rotation spread.Source: B. Target-Conditioned Finite-Window… · p. 5
  • Predicted aperture openings use model support width over a tactile window, adjusted for scale, yaw, window shift, scan-top offset, reflection, and aperture bias.Residuals use a weighted Huber RMS and include width, coverage, and aperture-bias terms.Source: B. Target-Conditioned Finite-Window… · p. 5
  • The matcher compares observed aperture widths and local nonplanarity with model predictions, combining coupled and independent sequence-fit losses with a scale prior.The depth-loss weight decreases as model anisotropy increases because width profiles are already informative for anisotropic targets.Source: B. Target-Conditioned Finite-Window… · pp. 5–6

C. Ranking and Target Grasping

The matcher ranks touched objects against the target model, then accepts a candidate only when fit, margin, and tactile evidence are sufficient. The selected object is centered and oriented for a force-limited grasp and lift.

C. Ranking and Target Grasping

  • The matcher ranks all touched physical TouchSets against the fixed target mesh using matching loss.The grasp center is corrected by the local contact center, and orientation is initialized from the global major direction.Source: C. Ranking and Target Grasping · p. 6
  • A candidate is accepted only when its first–second loss margin, absolute fit quality, and tactile evidence are sufficient.Source: C. Ranking and Target Grasping · p. 6
  • The gripper closes under the same force limit before lifting the selected object.The grasp uses the corrected local contact center and orientation initialized from the global major direction.Source: C. Ranking and Target Grasping · p. 6

A. Simulation Benchmark

The five-object simulation benchmark reports 84.0% retrieval, while a random-descent baseline completes all five discoveries less often than coverage-driven search. Separate validation reports 90.0% Top-1 accuracy for the local matcher.

A. Simulation Benchmark

  • 84.0% of targets were retrieved in the 100-trial five-object simulation benchmark.Apple was retrieved in 20/20 trials (100.0%), while Train was retrieved in 14/20 (70.0%).Source: A. Simulation Benchmark · p. 6, Table II · p. 6
  • At a 5 m budget, complete discovery occurred in 55% of layouts with coverage-driven search versus 30% with random descent.Random descent discovered fewer objects at every tested budget.Source: A. Simulation Benchmark · p. 6, Table I · p. 6

B. Local Tactile Matching Validation

C. Real-Robot Experiments

GLoTouch uses a parallel gripper for global probe-based search and local tactile recognition, then retrieves model-specified objects without external vision. Across five-object scenes, retrieval reaches 84.0% in simulation and 76.0% on hardware, with remaining failures tied to Train/Truck discrimination and physical stability.

C. Real-Robot Experiments

  • Real trials use five-object scenes with randomized poses, a supplied target STL and scene cardinality, and frozen parameters across objects and scenes.Source: C. Real-Robot Experiments · p. 7
  • On hardware, GLoTouch retrieves the specified target in 38 of 50 trials; Apple reaches 100.0% retrieval, while Train reaches 50.0%.Wukong, Nailong, and Truck achieve 70.0–80.0%.Source: C. Real-Robot Experiments · p. 7, Table V · p. 7
  • Train-to-Truck misrecognition and object rotation or toppling during tactile acquisition and clamping account for remaining failures.The authors link these failures to local geometric discrimination and physical manipulation stability.Source: C. Real-Robot Experiments · p. 7
  • Figure 6 shows probe-based global search, probe return and local recognition, then target retrieval continuing through a temporary lights-off period.Source: C. Real-Robot Experiments · p. 7, Figure 6 · p. 7

VII. CONCLUSION

  • The same gripper uses a passive probe for coverage-driven force search and bilateral visuotactile sensors for multi-height local contacts matched directly to a target STL.The local matcher uses finite-window support geometry and requires no object-specific training.Source: §VII · p. 7
  • In five-object scenes, retrieval reaches 84.0% (84/100) in simulation and 76.0% (38/50) on hardware.Source: §VII · p. 8
  • The component-level matcher achieves 90.0% Top-1 accuracy over 60 target/phase samples, while end-to-end retrieval reaches 84.0% (84/100) in simulation and 76.0% (38/50) on hardware in five-object scenes.Source: §VII · p. 8
  • The method assumes a known target STL, top-accessible objects, and a registered container frame; its coarse contour model uses round and box primitives.Source: §VII · p. 8
Loading 2609.27695v1…