Tactile Reconstruction of Contact Task Frames and Forces for Hybrid Force/Motion Control

I. INTRODUCTION

• Existing tactile pose and force methods do not jointly provide the contact variables needed to define force- and motion-controlled directions in hybrid force/motion control.Force regression alone does not provide the local task frame, while tactile servoing typically regulates pose or deformation rather than assigning contact force within a hybrid decomposition.Source: §I · p. 2
• For locally planar, featureless contact with negligible moment, tactile images encode indentation, surface-to-sensor tilt, and 3D contact force, but not contact position or tilt direction around the plane normal.The method therefore targets observable contact quantities rather than an unobservable full contact pose.Source: §I · p. 2
• A single soft optical tactile image yields those contact measurements with per-sample uncertainty, while self-labeling uses manipulator-imposed contacts and auxiliary F/T ground truth.The auxiliary sensor supplies offline labels and independent experimental comparison, not control-loop feedback.Source: §I · p. 2
• The tactile measurement and robot proprioception feed a state estimator that updates the contact plane, task frame, and interaction force during contact.The estimated surface geometry is retained with increasing uncertainty during brief contact breaks.Source: §I · p. 2
• Experiments regulate three contact-force components against a rigid board moved linearly and angularly by a human, using touch as the controller’s only exteroceptive feedback.The sensor orientation adapts to the reconstructed contact frame while tangential force components are driven to zero and normal force to a nonzero reference.Source: §I · p. 2

II. PROBLEM SETTING AND DESIGN RATIONALE

• Locally planar, featureless contact makes indentation depth and two tilt angles observable, but not tangential position or twist; negligible contact moment excludes moment regulation.The adopted assumptions also restrict the measurement model to three force components.Source: §II · p. 3
• The controller assigns forces along three contact-frame directions and two angular motions to align the sensor with the surface normal, with task augmentation completing the remaining twist.The task-space decompositions induced by these assumptions are summarized in Table I.Source: §II · p. 3, Table I · p. 3

III. SELF-LABELING DATASET ACQUISITION

• The acquisition protocol excites observable contact variables while varying unobservable ones, using robot proprioception for geometric labels and an auxiliary F/T sensor for force labels.The F/T sensor is used for data acquisition, not as an input to the regressor or online control loop.Source: §III · p. 3, A. Physical Setup · p. 3, Figure 2 · p. 4

A. Physical Setup

• A DigiTac sensor on a UR10 contacts a rigid, flat, horizontal metal plate; the dataset spans three replaceable sensor skins.The horizontal plate simplifies pose expression, and the protocol can be rotated for another known, fixed surface pose.Source: A. Physical Setup · pp. 3–4

B. Sampling Design

• Observable indentation, tilt, and force quantities are sampled while unobservable tangential position and normal twist are randomized and omitted from labels.This design aims to reduce acquisition-specific correlations learned by the regressor.Source: B. Sampling Design · p. 4
• The target pose varies indentation through depth, excites tangential forces through lateral displacement, and samples roll, pitch, and yaw to vary tilt and unobservable twist.The tool orientation is retained only when its z-axis lies within an admissible cone around the surface normal.Source: B. Sampling Design · p. 4
• The admissible tilt cone narrows with indentation to exclude contacts whose patch approaches the sensor rim.Phases 3–5 run below the UR10 maximum speed to limit impact and inertial effects.Source: B. Sampling Design · p. 4, C. Acquisition Cycle · p. 5
• Each sample independently draws a target pose and pre-contact orientation, then unloads the pad between samples to reduce sequential dependence from persistent deformation.The protocol does not assume consecutive samples are statistically uncorrelated.Source: B. Sampling Design · p. 4

C. Acquisition Cycle

• The six-phase cycle records an out-of-contact F/T reading before contact and the tactile image, robot pose, and F/T reading after settled indentation, shear, and post-rotation.Contact is released by retracting along the surface normal without commanded in-plane dragging.Source: C. Acquisition Cycle · pp. 4–5, Figure 3 · p. 5

D. Label Extraction

• Geometric labels use the attained pose relative to the plate, while force labels use F/T readings separated into contact force, tool weight, and slowly varying bias.An out-of-contact measurement provides the reference for separating these contributions.Source: D. Label Extraction · p. 5
• Assuming F/T bias stays approximately constant over one acquisition cycle, subtracting the out-of-contact reading from the contact reading cancels that bias.The resulting force is used as the sample label.Source: D. Label Extraction · p. 5

IV. TACTILE MEASUREMENT REGRESSOR

• Grayscale downsampling reduces image-processing cost, while convolution is selected to model the spatially structured deformation field encoded by pin displacement.Training, evaluation, and later control use the same image format.Source: D. Label Extraction · p. 5, §IV · p. 5

A. Architecture

• The ResNet-style architecture combines a residual convolutional backbone and squeeze-and-excitation blocks with parallel MLP heads for the six-dimensional measurement mean and log-variance.Figure 4 depicts this architecture; the paired outputs provide an estimate and per-sample confidence for Bayesian filtering.Source: A. Architecture · pp. 5–6, Figure 4 · p. 6
• The final backbone uses a shallow stem and four residual stages, with downsampling and adaptive average pooling to produce a 128 × 8 × 6 feature map.The backbone is deliberately kept small to support real-time deployment.Source: A. Architecture · p. 6
• Each residual block combines two convolutions with a skip path, while a lightweight squeeze-and-excitation block recalibrates features channel-wise before residual addition.The skip path is an identity mapping or a 1 × 1 convolution when channel expansion is needed.Source: A. Architecture · p. 6
• Two parallel MLP heads predict the measurement mean and per-sample log-variance, which is exponentiated during inference to construct the measurement covariance.Architecture modifications were retained when they reduced validation loss, but the sequential comparisons were not reported as an ablation study.Source: A. Architecture · pp. 5–6

B. Training Methodology

• Training uses 10,000 labeled samples randomly partitioned into 70% training, 15% validation, and 15% test subsets, reserving the test set for final evaluation after configuration selection.Source: B. Training Methodology · p. 6
• The regressor is trained by maximum-likelihood estimation under a diagonal Gaussian observation model, using Gaussian negative log-likelihood for each output dimension.Source: B. Training Methodology · p. 6
• Homoscedastic pretraining optimizes a shared global log-variance while warming up the mean, addressing convergence failures observed when both outputs are optimized from initialization.Source: B. Training Methodology · p. 6
• Heteroscedastic fine-tuning starts from the Stage 1 early-stopping checkpoint and learns input-dependent, per-sample uncertainty estimates by optimizing all parameters end-to-end.Source: B. Training Methodology · p. 6

V. TACTILE-DRIVEN HYBRID FORCE/MOTION CONTROL

• The control objective is force regulation against an unknown, slowly time-varying surface while maintaining sensor alignment with its local normal, using touch as the only exteroceptive feedback.Source: §V · p. 7
• The system fuses tactile measurements with robot proprioception in an EKF, then combines estimated contact variables and hybrid targets through an admittance layer to generate smooth motion references.Source: §V · p. 7

• The estimator fuses instantaneous tactile measurements with tool pose to represent surface geometry and interaction force continuously in a frame suitable for decoupled control.Source: A. Estimation of the Contact State · p. 7, Figure 5 · p. 8
• The state retains plane offset and normal plus three contact-force components, but excludes tangential contact coordinates, rotation about the surface normal, and contact moment.Source: A. Estimation of the Contact State · p. 7
• The contact state follows a random-walk process, while per-image tactile covariance lets the EKF weight each measurement according to its uncertainty.Source: A. Estimation of the Contact State · pp. 7–8
• The measurement model predicts indentation, two surface-to-sensor tilt angles, and tool-frame contact force from the persistent world-frame contact state.Source: A. Estimation of the Contact State · p. 8
• A Schmitt trigger on regressed indentation detects contact, and hysteresis suppresses repeated switching near contact onset and detachment.Source: A. Estimation of the Contact State · p. 8
• When contact is absent, the EKF skips measurement updates and propagates the state, preserving the latest surface estimate while uncertainty increases through process noise.Source: A. Estimation of the Contact State · p. 8

• The implemented task decomposition assigns three translational directions to force regulation and two tilt directions to motion regulation; normal loading sets the friction cone for tangential force commands.Source: B. Control Pipeline · p. 8
• The tactile estimate resolves surface normal and two tilt angles, but not twist about the normal; the commanded twist is therefore arbitrary rather than tactile-regulated.Source: B. Control Pipeline · pp. 8–9
• Commanded force must remain within the friction cone, and commanded tilt must satisfy the approach open-half-space restriction used during acquisition.Source: B. Control Pipeline · p. 9
• Two complementary admittance channels map hybrid targets through the estimated contact frame into translational and rotational motion references.Source: B. Control Pipeline · p. 9
• The translational position admittance uses virtual inertia and damping, with its state expressed as velocity in the estimated contact frame and resulting references expressed in the world frame.Source: B. Control Pipeline · p. 9
• The rotational channel leaves twist about the estimated normal free, choosing it by task augmentation, and filters the target orientation into a smooth reference without driving the filter from contact force.Source: B. Control Pipeline · p. 9
• The UR10 tracks the resulting references with Cartesian velocity commands formed from reference velocities and position and orientation error feedback.Source: B. Control Pipeline · p. 9

• The section evaluates the regressor on a held-out test set and the integrated estimation-and-control system in a dynamic contact experiment.Source: §VI · p. 9
• Training and validation losses are shown for homoscedastic pretraining and heteroscedastic fine-tuning from the Stage 1 early-stopping checkpoint.Source: Figure 6 · p. 10

A. Regressor Performance

• Heteroscedastic fine-tuning reduced validation loss from −2.035 at the Stage 1 checkpoint to −2.348.Training required 341 Stage 1 epochs and 94 Stage 2 epochs, approximately 2 h total on one NVIDIA Tesla T4 GPU.Source: A. Regressor Performance · p. 10
• 0.175 mm indentation RMSE; tilt RMSEs were 1.767 deg and 1.993 deg, while x/y/z force RMSEs were 0.051/0.054/0.206 N.The indentation error is comparable in scale to the manipulator's 0.1 mm repeatability, which does not separate regressor error from label uncertainty.Source: A. Regressor Performance · p. 10
• Adding torque-based consistency supervision produced no measurable improvement in lateral-force accuracy over Eq. (12) alone.The torque channels offered redundant force information under Assumption 2, but this term was not part of the reported model.Source: A. Regressor Performance · p. 10
• Across all six outputs, empirical error generally increased with predicted standard deviation, with binned curves close to identity over most ranges.Larger deviations and variability occurred for orientation channels; the results support using predicted variance as EKF measurement covariance within the tested acquisition domain.Source: A. Regressor Performance · p. 10

B. Closed-Loop Control Experiment

• The feasibility trial used 20 Hz Cartesian commands under Assumptions 1–2 and was not an exhaustive sweep of materials, geometries, friction, or force references.An operator continuously translated and rotated a rigid plywood board while the hybrid target remained constant.Source: B. Closed-Loop Control Experiment · p. 10
• The estimated normal force approached its 2.5 N reference while tangential components remained around zero; bounded signals maintained regulation throughout the reported trial.Deviations occurred principally during board-pose changes as the estimated frame, admittance dynamics, and manipulator responded.Source: B. Closed-Loop Control Experiment · p. 11
• The tactile force prediction and offline F/T reconstruction showed similar temporal evolution, with largest discrepancies during force excursions and contact transients.F/T measurements were used only for evaluation, not as inputs to the regressor, EKF, target generator, or controller.Source: B. Closed-Loop Control Experiment · p. 11
• As the board orientation varied, the target orientation followed the EKF-reconstructed task frame, and measured tool orientation followed the filtered reference.The orientation channel continuously adapted sensor alignment to estimated surface tilt.Source: B. Closed-Loop Control Experiment · p. 11
• Without retraining, the same regressor showed qualitatively comparable behavior on a different robot, vertical plastic panel, laboratory, and skin; those tasks are undocumented.The trial was included as visual evidence of operation outside the principal setup.Source: B. Closed-Loop Control Experiment · p. 11

VII. CONCLUSIONS

• A ResNet-style regressor maps each tactile image to indentation depth, two surface-to-sensor tilt angles, and 3D contact force, with per-sample uncertainty.An auxiliary force/torque sensor supplies offline ground-truth labels and an independent validation reference.Source: §VII · pp. 11–12
• The system regulated contact force against a flat, rigid board in human-imposed linear and angular motion, using touch as its only exteroceptive feedback.The plots mark contact phases detected from regressed indentation depth.Source: Figure 8 · p. 12, §VII · p. 12
• An EKF fuses tactile measurements with robot proprioception to estimate the contact task frame and interaction force for an admittance-based hybrid force/motion controller.Source: §VII · p. 12
• The trained regressor showed qualitatively comparable behavior without retraining on a different manipulator, plastic panel, and laboratory setup.These trials used a setup different from the one in Tab. II.Source: §VII · p. 12, Figure 9 · p. 13
• The results are scoped to rigid, planar, featureless surfaces, moderate indentation, and the prescribed contact cone; generalization to curved, textured, or compliant surfaces or larger shear-to-normal force ratios is not claimed.Source: §VII · p. 13
• Rotation about the surface normal and tangential contact position are unobservable under the adopted featureless-plane contact model.Normal-axis rotation is currently fixed through arbitrary task augmentation; tangential motion may still leave signatures through shear and incipient slip.Source: §VII · p. 13
• Floating-base deployment would require fusion with its onboard state estimator and an interaction controller that accounts for coupled base and robot dynamics.The reconstructed contact variables are not specific to fixed-base manipulators.Source: §VII · p. 13