PaperlayerView exact revision on arXiv

Source-linked AI summary

A Minimal Optical-Flow Representation for Vision-Based Tactile Rotation Classification in Robotic Manipulation Across Gravity Domains

Oscar Martinez-Bernal, Mario Cavero-Vidal, Francesco Grella, Carol Martinez

arXiv:2610.12073v1cs.ROcs.CV

TL;DR

Earth-developed tactile perception may not transfer directly to reduced-gravity settings, motivating tests of compact motion features across gravity domains. The study classifies load-induced rotation from aggregated optical flow and finds that a compact representation performs across the evaluated gravities, while Earth-only training degrades in orbit.

  • Problem

    Tactile methods are typically developed and validated under Earth gravity, leaving their transfer to reduced-gravity conditions uncertain for space robotics.

    Source: §I · p. 1

  • Method

    The study aggregates dense optical flow from simulated tactile frames into a compact representation and evaluates rotation-direction classifiers trained on Earth-only or four-gravity data.

    Source: E. Classifiers and Training · p. 4, §VII · p. 7

  • Results

    0.963 accuracy is reported for the 126-feature representation, while Earth-only XGBoost accuracy falls from 0.944 on Earth to 0.759 in orbit.

    Source: B. Earth-Only Training · p. 5, §VII · p. 7

  • Takeaways & Limitations

    Earth-gravity performance alone is insufficient to establish transferability, and the 40-feature representation retains 0.957 accuracy across the evaluated gravity conditions.

    Source: §VII · p. 7

    The study uses simulation with a rigid gel model, and sensor-orientation changes require a consistent image reference frame or orientation-diverse training data.

    Source: D. Limitations · p. 7

Abstract

from arXiv · show

Vision-based tactile sensors provide rich contact information, but processing high-resolution images can be costly for resource-constrained platforms such as space robots. This work investigates whether a compact representation of tactile motion can classify object rotation across different gravity conditions. Dense optical flow from a simulated GelSight Mini is aggregated over a 7x9 grid into 126 features and used to classify the direction of load-induced rotation under Earth, Mars, Moon, and orbital gravity. Gravity causes a small but significant shift in these features, accounting for 1.6% of their variance (R2 = 0.016). Despite its small magnitude, this shift affects models trained only on Earth data: XGBoost accuracy decreases from 94.4% on Earth to 75.9% in orbit. In contrast, a single model trained across all four gravity domains achieves 96.3% overall accuracy and 95.1%-97.0% across individual domains, without using gravity as an input. The representation can also be reduced to 40 features while retaining 95.7% accuracy, with XGBoost requiring only 0.14 ms per inference. These findings show that Earth-gravity performance alone is insufficient to establish the transferability of tactile perception for space robotic manipulation, highlighting the need to account for gravity-induced domain shifts during training and validation.

I. INTRODUCTION

Vision-based tactile sensing can provide contact information for space robots, but processing its images is costly and Earth-developed methods may not transfer to other gravity conditions. This work tests a compact tactile-motion representation for rotation classification across gravity domains.

Motivation

  • Tactile images provide rich contact information, but processing them directly can be costly for resource-constrained platforms such as space robots.Optical flow captures tactile motion in a reduced feature set and has been used for lightweight classification tasks.Source: §I · p. 1
  • Earth-only tactile classifiers may not transfer directly to reduced-gravity environments, which are difficult to reproduce and validate experimentally on Earth.Gravity can alter object dynamics and, in turn, the tactile signals observed during manipulation.Source: §I · p. 1

Research questions

  • The study asks whether compact tactile motion can classify manipulation events and whether performance transfers across gravity conditions.Load-induced rotation of a grasped object is the controlled interaction event; its direction can inform corrective manipulation before grasp stability is compromised.Source: §I · p. 1

Approach

  • Dense optical flow from consecutive simulated GelSight Mini frames is aggregated over a fixed grid for lightweight classification across Earth, Mars, Moon, and orbital gravity.Gravity is not supplied as a classifier input.Source: §I · pp. 1–2

Contributions

  • The analysis quantifies gravity-induced feature shifts, tests Earth-only transfer, then evaluates all-gravity training and feature reduction.The contributions include a 40-feature representation that retains 95.7% accuracy, less than one percentage point below the full 126-feature representation.Source: §I · p. 2
  • A single model trained across Earth, Mars, Moon, and orbital gravity achieves consistent classification without gravity as an input.The study also evaluates whether the representation can be reduced while preserving classification performance.Source: §I · p. 2

II. RELATED WORK

Related work covers vision-based tactile sensing and simulation, positioning this study as an evaluation of compact optical-flow features for rotation classification across gravity conditions.

A. Vision-Based Tactile Sensing

  • Optical flow has supported contact-shape, force, geometry, depth, and point-cloud estimation, while this study uses a compact representation to classify load-induced rotation across gravity conditions.Prior examples include DelTact, GelFlow, and 3D contact point clouds.Source: A. Vision-Based Tactile Sensing · p. 2

B. Tactile Simulation

  • Tactile simulators render contact images, and sim-to-real transfer is not automatic; this study uses TacEx with GelSight Mini in Isaac Sim to examine gravity rather than the sim-to-real gap.Only grayscale images are used in the PhysX rigid-body configuration.Source: B. Tactile Simulation · p. 2

C. Tactile Sensing in Space Robotics

The study addresses a measured gap in space robotics: whether gravity changes the tactile features used for classification. It evaluates transfer using a controlled simulated task and a compact optical-flow pipeline.

C. Tactile Sensing in Space Robotics

  • Prior space-robotics work had not measured gravity’s effect on tactile signals or classifier features; this study examines that gap with one representation and one controlled task.The study asks whether rotation-direction classification generalizes across gravities without richer tactile images, marker displacement, reconstructed geometry, learned force maps, robot state, or gravity input.Source: C. Tactile Sensing in Space Robotics · p. 2

A. Strategy Overview

  • The pipeline converts simulated tactile frames to dense optical flow, aggregates it into 126 features, and classifies load-induced rotation direction.It uses a fixed grid and lightweight classifiers.Source: A. Strategy Overview · p. 2, Figure 2 · p. 3

B. Optical Flow Representation

  • Dense optical flow is computed between consecutive grayscale tactile frames, then reduced to median horizontal and vertical flow components in each of 63 grid cells.The 7-by-9 grid yields 126 features: 63 dx values and 63 dy values.Source: B. Optical Flow Representation · p. 3

C. Rotation-Direction Target

Rotation direction is the classification target, with labels assigned to contact-pattern motion and flow fields from both sensor pads used for training.

C. Rotation-Direction Target

  • Each event’s label is the clockwise or counterclockwise direction of its contact-pattern rotation, and every flow field from that event receives that label.Because the gripper pads face the object from opposite sides, the same load produces opposite rotation directions on the two sides.Source: C. Rotation-Direction Target · p. 3
  • The classifier receives only optical-flow features, excluding force magnitude, load position, gravity, repetition, sensor side, and frame index.Both sensors are used for training, with each sensor treated as an independent sample.Source: C. Rotation-Direction Target · p. 3

D. Feature Selection

The study selects compact optical-flow features and evaluates four lightweight classifiers using trial-separated training and test sets. It compares Earth-only training with training across all four gravity conditions.

D. Feature Selection

  • The 126-dimensional representation uses horizontal and vertical flow medians from 63 grid cells; two selection methods rank original features without transforming classifier inputs.PCA-guided selection ranks features by contribution to components explaining 90% of training variance, while mRMR ranks target relevance while penalising redundancy.Source: D. Feature Selection · p. 3
  • A grid-resolution ablation tests increasingly compact spatial representations and the contribution of individual flow components.Both feature-selection methods use the training set only.Source: D. Feature Selection · p. 3

E. Classifiers and Training

  • The four evaluated classifiers are XGBoost, random forest, KNN, and logistic regression, with logistic regression serving as a linear reference.KNN uses standardised features.Source: E. Classifiers and Training · pp. 3–4
  • An 80/20 trial-level, class-stratified split keeps events from the same trial in a single partition.The Earth regime trains only on Earth data; the All regime includes all four gravity conditions, with grouped 3-fold cross-validation for hyperparameter selection of XGBoost, random forest, and KNN.Source: E. Classifiers and Training · p. 4
  • Accuracy per flow field is the primary metric, alongside balanced accuracy, macro-F1, ROC-AUC, and PR-AUC.Final models are fitted on the complete training set and evaluated once on the held-out test set.Source: E. Classifiers and Training · p. 4

IV. EXPERIMENTAL SETUP

The experiments simulate a grasped profile under varied loads and gravity conditions, recording tactile events for rotation-direction classification. The resulting dataset is partitioned for the two training regimes.

A. Simulation and Load Application

  • The simulated setup uses a Franka arm with two GelSight Mini sensors gripping a horizontal metallic profile while an external vertical force produces torque.The force is applied at a distance from the grasp, about an axis parallel to the sensor optical axis.Source: A. Simulation and Load Application · p. 4, Figure 3 · p. 4
  • The design varies force magnitude, direction, moment arm, repetitions, and gravity across Earth, Mars, Moon, and orbit (0g).It uses 3 force magnitudes, 2 directions, 20 positions, and 4 repetitions in each of 4 gravity conditions; the simulator is reset between trials.Source: A. Simulation and Load Application · p. 4

B. Dataset and Partitions

  • Table II summarizes the event and flow-field partitions used by the Earth-only and all-gravity training regimes.The Earth regime uses Earth-gravity data, while the All regime includes all four gravity conditions.Source: Table II · p. 4, B. Dataset and Partitions · p. 4
  • The dataset contains 2366 events from 1245 trials, with 563–620 events per gravity condition and approximately balanced ccw and cw classes.Events dominated by linear slip were excluded; optical-flow fields from the rotating contact-pattern interval inherit the event label.Source: B. Dataset and Partitions · p. 4

V. RESULTS

Earth-only training shows declining accuracy on unseen gravity domains, while the feature analysis finds a small but measurable gravity-associated shift. The shift is statistically significant, but smaller than the effects of several other experimental factors.

A. Gravity-Domain Analysis

  • Gravity accounts for 1.6% of optical-flow representation variance (R2 = 0.0164), less than load position (R2 = 0.591), sensor side (0.189), force magnitude (0.055), and load direction (0.043).The PERMANOVA result is significant (pseudo-F = 13.12, p = 0.001; 999 permutations).Source: A. Gravity-Domain Analysis · p. 4, Figure 4 · p. 5
  • FDR-corrected tests distinguish Earth from Mars, Moon, and orbit (R2 = 0.017–0.020, p = 0.002), while Moon and orbit are not distinguished (R2 = 0.0008, p = 0.53).Gravity effects occur in 78 of 126 features; the largest cell-level effects are ϵ2 = 0.0595 for dy and 0.0579 for dx.Source: A. Gravity-Domain Analysis · p. 5
  • The feature analysis identifies a small but measurable gravity shift before testing its association with reduced classification performance under Earth-only training.The figure maps Kruskal–Wallis effect size by flow component and grid cell.Source: A. Gravity-Domain Analysis · p. 5, Figure 4 · p. 5

B. Earth-Only Training

  • Under Earth-only training, XGBoost accuracy is 0.944 on Earth and falls to 0.759 in orbit, a maximum decrease of 0.185; it is 0.858 on Mars and 0.817 on the Moon.Source: B. Earth-Only Training · p. 5
  • Earth-to-orbit accuracy decreases from 0.938 to 0.655 for random forest, from 0.934 to 0.591 for KNN, and from 0.857 to 0.688 for logistic regression.Accuracy declines progressively from Earth through Mars and Moon to orbit for all four classifiers, with the same pattern in balanced accuracy, macro-F1, ROC-AUC, and PR-AUC.Source: B. Earth-Only Training · p. 5

C. All-Gravity Training

Training with all four gravity domains yields high XGBoost accuracy overall and across individual domains without gravity as an input. The feature ablation finds that a 40-feature grid retains accuracy close to the full representation.

C. All-Gravity Training

  • XGBoost reaches 0.963 accuracy on the all-gravity held-out test set; Fig. 5 compares accuracy across gravity domains and classifiers.Random forest reaches 0.955, KNN 0.949, and logistic regression 0.857; XGBoost ROC-AUC and PR-AUC are both 0.996.Source: C. All-Gravity Training · p. 5, Figure 5 · p. 5, Table IV · p. 6
  • XGBoost accuracy ranges from 0.951 on Mars to 0.970 on Earth, with 0.968 on the Moon and 0.966 in orbit, without gravity as a classifier input.Similar performance is observed across load magnitudes, sensor sides, and rotation magnitudes.Source: C. All-Gravity Training · p. 5, Figure 5 · p. 5

D. Minimal Representation

  • A 4 × 5 grid with 40 features reaches 0.957 accuracy, compared with 0.963 for the full 7 × 9 grid with 126 features.PCA-guided selection reaches 0.963 with 74 features; more aggressive reductions reach 0.939 with 14 mRMR features, 0.925 with a 2 × 3 grid, and 0.670 with a single-cell representation.Source: D. Minimal Representation · p. 5
  • Using only dx or dy reduces accuracy to 0.944 or 0.947, respectively.The 4 × 5 grid is the most compact spatial representation evaluated while remaining within one percentage point of the full representation.Source: D. Minimal Representation · p. 5

E. Computational Cost

Gravity explains only a small share of optical-flow feature variance, yet Earth-only classifiers lose accuracy in reduced gravity. Dense optical flow, rather than XGBoost inference, dominates computational cost in the reported setup.

E. Computational Cost

  • 10.15 ms for Farnebäck optical flow and 1.43 ms for grid aggregation exceed XGBoost’s 0.14 ms inference time per flow field.Runtime was measured on one CPU thread and is machine-dependent.Source: E. Computational Cost · p. 6

A. Gravity-Induced Domain Shift

  • 1.6% of feature variance is attributed to gravity, yet Earth-only XGBoost accuracy falls from 0.944 on Earth to 0.759 in orbit, with degradation increasing from Mars to Moon and orbit.The shift is small in overall variance but measurable and associated with substantial transfer degradation.Source: A. Gravity-Induced Domain Shift · p. 6

B. Generalisation Across Gravity Domains

Training on all four gravity domains yields consistent XGBoost performance without gravity as an input. Earth-only performance loss varies by classifier, and XGBoost is more robust to the shift than the other tested classifiers.

B. Generalisation Across Gravity Domains

  • XGBoost achieves 0.963 overall accuracy, with per-gravity accuracy ranging from 0.951 to 0.970, when trained across all four gravity domains.Gravity is not provided as an input.Source: B. Generalisation Across Gravity Domains · p. 6
  • The all-gravity model achieves consistent performance across the four evaluated domains despite not receiving gravity as an input.This is the reported condition for the cross-gravity result.Source: B. Generalisation Across Gravity Domains · p. 6
  • Maximum Earth-only accuracy loss ranges from 0.169 for logistic regression to 0.342 for KNN, while XGBoost combines the highest accuracy with greater robustness.Cross-gravity performance depends on both the tactile representation and the classifier.Source: B. Generalisation Across Gravity Domains · pp. 6–7

C. Minimal Representation and Cost

The 126-feature representation achieves 0.963 accuracy, while a 40-feature version retains 0.957; dense optical-flow computation remains the main computational cost. The findings are bounded by simulation with a rigid gel model and sensitivity to sensor orientation.

C. Minimal Representation and Cost

  • A 4×5 grid reduces the feature set from 126 to 40 with accuracy decreasing only from 0.963 to 0.957; more aggressive compression yields 0.939 with 14 features and 0.670 for one cell.Farnebäck optical flow and grid aggregation require approximately 11.6 ms per frame pair, compared with 0.14 ms for XGBoost inference.Source: C. Minimal Representation and Cost · p. 7

D. Limitations

  • The findings are limited to simulation with a rigid gel model, and classifier sensitivity to sensor orientation constrains deployment across orientations.The paper says different orientations require a consistent image reference frame or orientation-diverse training data.Source: D. Limitations · p. 7

VII. CONCLUSION AND FUTURE WORK

  • 0.963 accuracy with 126 features decreases to 0.957 with a 40-feature grid, while XGBoost inference takes 0.14 ms and dense optical flow remains the main cost.The conclusion reports the overall representation results and identifies dense optical-flow computation as the main computational cost.Source: §VII · p. 7
  • Earth-gravity performance alone is insufficient to establish transferability; future work targets real-sensor validation, unseen gravity conditions, and broader contact events, loading conditions, and object geometries.The paper concludes that compact representations can preserve most classification performance across the evaluated gravity conditions.Source: §VII · p. 7
Loading 2610.12073v1…