PaperlayerView exact revision on arXiv

Source-linked AI summary

Active Clothing Material Perception using Tactile Sensing and Deep Learning

Wenzhen Yuan, Yuchen Mo, Shaoxiong Wang, Edward Adelson

arXiv:1711.00574v2cs.RO

TL;DR

The paper addresses how robots can perceive fine-grained properties of clothing through touch, rather than relying only on rough visual categories. It builds a Kinect-guided, GelSight-based system with CNNs for property recognition and exploration selection, and reports autonomous closed-loop exploration of known and unknown clothes, while noting overfitting and limited category counts.

  • Problem

    Fine-grained clothing perception is difficult because robots must obtain adequate tactile information and generate effective exploration procedures for varied clothes.

  • Method

    The system uses Kinect depth images to select wrinkle locations, GelSight squeezing to collect tactile images, and CNNs to classify 11 clothing properties and guide closed-loop exploration.

  • Results

    The robot recognizes clothing properties for seen and unseen items and improves online classification precision with re-trials, averaging 1.71 trials per exploration.

  • Takeaways & Limitations

    The framework supports autonomous tactile exploration of common clothes and a relatively comprehensive representation of their physical and semantic properties.

Abstract

from arXiv · show

Humans represent and discriminate the objects in the same category using their properties, and an intelligent robot should be able to do the same. In this paper, we build a robot system that can autonomously perceive the object properties through touch. We work on the common object category of clothing. The robot moves under the guidance of an external Kinect sensor, and squeezes the clothes with a GelSight tactile sensor, then it recognizes the 11 properties of the clothing according to the tactile data. Those properties include the physical properties, like thickness, fuzziness, softness and durability, and semantic properties, like wearing season and preferred washing methods. We collect a dataset of 153 varied pieces of clothes, and conduct 6616 robot exploring iterations on them. To extract the useful information from the high-dimensional sensory output, we applied Convolutional Neural Networks (CNN) on the tactile data for recognizing the clothing properties, and on the Kinect depth images for selecting exploration locations. Experiments show that using the trained neural networks, the robot can autonomously explore the unknown clothes and learn their properties. This work proposes a new framework for active tactile perception system with vision-touch system, and has potential to enable robots to help humans with varied clothing related housework.

I. INTRODUCTION

The paper develops an autonomous vision-touch system for perceiving fine-grained clothing properties, addressing the challenges of extracting useful tactile information and selecting effective exploration procedures. It combines GelSight squeezing, Kinect-guided exploration, and neural networks to recognize physical and semantic properties and support clothing-related housework.

  • Clothing properties such as thickness, fuzziness, smoothness, softness, wearing season, and washing method help robots manage, maintain, sort, and organize clothes.
  • Fine-grained property perception remains challenging because robots must discriminate subtle differences within clothing while obtaining adequate tactile information and generating effective exploration procedures.
  • The system squeezes clothing, mostly at wrinkles, with a GelSight sensor and trains a CNN for multi-label property classification.
  • A second CNN uses Kinect depth images to select wrinkle locations, while closed-loop exploration repeats data collection when tactile predictions lack confidence.
  • 153 clothing pieces spanning different types, materials, and sizes support recognition of 11 physical and semantic properties on common clothes.

II. RELATED WORK

Prior clothing research largely focuses on manipulation or vision-based recognition of rough clothing types. Fine-grained recognition of properties across varied common clothes remains an open challenge.

  • Existing clothing studies mainly address grasping, folding, unfolding, or vision-based classification into rough types such as pants, t-shirts, and coats.
  • A multimodal system combined RGBD shape, photometric-stereo texture, and fingertip force sensing, improving material recognition, but fine-grained common-clothing properties remain challenging.

B. Tactile sensors and GelSight

Tactile sensing provides force, contact, shape, and material information, while GelSight offers high-resolution optical measurements of surface geometry and texture. CNNs have also been applied to GelSight images for material-property estimation.

  • Most tactile sensors measure force or contact distribution over an area, supporting object-shape classification and material-property estimation with pressing or sliding motions.
  • GelSight measures surface geometry at around 30 micros spatial resolution and uses printed markers to measure contact force or shear.
  • GelSight’s high resolution makes surface texture effective for discriminating material categories.
  • CNN-based tactile methods have estimated object hardness and fabric properties from GelSight images using networks pretrained on ordinary images.

III. DATA COLLECTION

The study constructs a broad clothing dataset and labels garments with common human-used properties for tactile perception. Data collection covers varied everyday items and supports binary or multiclass property labels.

  • The robot explores clothes by squeezing a part while GelSight records a sequence of tactile images, with Kinect guiding motion planning.
  • 153 new and second-hand clothing items span everyday types, materials, and sizes, with a small number of additional fabric products.
  • The dataset uses 11 common clothing-property labels, represented as either binary or multiple classes.
  • Table I presents the clothing-property labels and example classes used for annotation.

B. Robotic System Setup

The robotic system combines a robot arm, gripper, GelSight tactile sensor, and RGBD camera for autonomous clothing exploration. Kinect depth processing selects wrinkle-based gripping points, while GelSight validates contact and records tactile data.

  • Hardware setup: The hardware comprises a robot arm, parallel gripper, GelSight tactile sensor, and external RGBD camera.The GelSight sensor is mounted as one gripper finger, while the camera guides exploration.
  • Operational caveat: Repeated large shear forces wear the GelSight surface, requiring elastomer replacement and introducing differences between tactile images.The manually made elastomers also vary slightly in marker patterns and shape.
  • Autonomous collection: The robot autonomously collects training data through a flow-charted exploration process.The supplied passage identifies Figure 4 as the flow chart for autonomous data collection.
  • Gripping-point selection: Kinect depth images are transformed into the world frame so clothing height above the table can be analyzed.The transformed depth map represents clothing height through its z value.
  • Gripping-point selection: Laplacian processing and three pyramid levels identify wrinkle points across different wrinkle widths, from which one target is randomly selected.The gripper orientation is calculated from the wrinkle direction before gripping.
  • Tactile acquisition: The gripper closes at 5mm/s while GelSight records 10 to 25 frames, then contact validity is judged from the tactile images.No-contact iterations are marked invalid and the gripping location is labeled a failure case.

IV. CLOTHES CLASSIFICATION USING DEEP LEARNING

The system separates autonomous exploration-point selection from clothing-property estimation, using distinct neural networks for these two goals.

  • System design: Two separate neural networks select clothing exploration points and estimate properties from collected tactile data.The design assigns one network to each independent goal.

A. Networks for property perception

Property perception uses CNN-based multi-label classification on GelSight images, with both single-frame and sequence-based inputs. The sequence model uses force-varying tactile frames processed through CNN and LSTM components.

  • Single-frame perception: CNNs independently classify multiple clothing-property labels from GelSight data.The labels correspond to clothing properties and are trained independently.
  • Single-frame perception: The single-frame network uses the GelSight image captured at maximum contact force.A single CNN classifies that selected image.
  • Sequence perception: The sequence model processes tactile images captured during increasing squeeze forces, which record surface shape and texture under different forces.The paper describes these image sequences as more informative than single images.
  • Sequence perception: The video network connects CNN features to an LSTM with a hidden state of 2048 dimensions.It uses fc6 features from VGG16 as the LSTM input.
  • Sequence perception: The sequence contains 9 frames as a balance between computational cost and information.Frames are sampled at equal time intervals until maximum contact.

B. Networks for gripping point selection

The gripping-point network predicts whether a candidate location will produce effective tactile data from cropped Kinect depth images. Training uses labeled exploration outcomes and CNN-based optimization procedures.

  • Gripping-point network: A VGG16-based Depth CNN classifies whether a cropped depth image represents a potentially successful gripping location.The gripping point is centered in the cropped world-frame depth image.
  • Data division: The dataset uses 123 clothing items for training and validation and 30 items for testing.For the 123 training and validation items, 85% of collecting iterations are used for training and 15% for validation.
  • Training labels: Exploration failures include no contact and unclear tactile images; only successful explorations train the tactile CNNs.Successful iterations are assigned class 1 for Depth CNN training.
  • Optimization: The networks use stochastic gradient descent, ImageNet-pretrained weights, and joint CNN-LSTM training for the video model.The video network is jointly trained for 500 epochs with a dropout rate of 0.5.
  • Augmentation: Data augmentation adds random intensity values during GelSight-image training.The sequence model also varies the input sequence slightly by timestamp.

D. Online robot test with re-trials

The robot uses tactile and visual neural networks to perceive clothing properties, but performance is constrained by data, sensor variation, model mismatch, and context-dependent labels.

  • Data and evaluation: 6616 exploration iterations produced 3762 valid GelSight videos for offline property-perception experiments.The remaining 2854 iterations lacked good data because of inadequate gripping locations.
  • Property perception: The tactile CNN predicts properties on seen and novel clothes with precision much better than chance, but validation–test gaps indicate overfitting.The supplied passages do not include Table II’s numerical precision values.
  • Limitations: The dataset contains only 2 to 5 clothes per refined category, limiting coverage despite its variety of clothing types.
  • Limitations: Five GelSight sensors introduce image differences through distinct optical properties.
  • Model design: CNNs designed for visual images may not be optimal for GelSight images, and VGG19 performs relatively better than tested AlexNet and VGG16 alternatives.
  • Temporal input: Video-trained networks do not significantly improve performance, possibly because texture overfitting and limited training data hinder learning from dynamic GelSight changes.

B. Exploring planning

A depth-image CNN selects effective gripping locations for exploration, achieving above-chance success on both familiar and unseen clothes but beginning to overfit.

  • Exploring planning: 0.73 success rate on both validation and unseen-clothes test sets exceeds the 0.5 chance baseline for gripping-location selection.
  • Exploring planning: The clothing item’s identification has limited influence on gripping-location selection performance.
  • Exploring planning: The network quickly reaches its best performance during training and then starts to overfit.

C. Online robotic test

The online test shows that autonomous tactile exploration can recognize clothing properties on unseen clothes, with re-trials improving confidence and precision despite raw sensor variation.

  • 77.42% of clothes required fewer than 2 grasps to obtain a confident property estimate.The remaining clothes required multiple explorations and their properties were still not well recognized.
  • 1.71 trials per exploration was the robot’s average during online testing.
  • Re-trials largely increased property-classification precision compared with accepting every initial exploration.The robot re-did exploration when the tactile CNN’s property estimate was not confident.
  • The system combines a tactile CNN for property classification with Kinect-guided exploration and a depth CNN for selecting contact locations.
Loading 1711.00574v2…