Source-linked AI summary
The Curious Robot: Learning Visual Representations via Physical Interactions
Lerrel Pinto, Dhiraj Gandhi, Yuanfeng Han, Yong-Lae Park, Abhinav Gupta
TL;DR
The paper asks whether physical interaction can provide supervision for visual representation learning instead of passive observations or category labels. It trains a shared ConvNet from Baxter-robot interactions across grasping, pushing, tactile sensing, and viewpoint observation, using more than 130K datapoints. The resulting representation improves classification over training from scratch and achieves 72% instance-level recall@1, compared with 69% for ImageNet.
Problem
Computer vision commonly learns visual features from category labels or passive observations, while biological visual learning can use physical interaction without supervision from other agents.
Method
A Baxter robot performs grasping, pushing, tactile sensing, and viewpoint observation, using the resulting supervision to train a shared ConvNet.
Results
72% instance-level recall@1 exceeds ImageNet’s 69% and randomNet’s 6%, while classification accuracy improves over training from scratch across the reported datasets.
Takeaways & Limitations
Physical interactions can provide supervision for learning a meaningful visual representation that transfers to classification and image retrieval evaluations.
Takeaways & Limitations
The paper defines “unsupervised” as lacking supervision from other agents, while allowing supervision from other modalities or time.
Abstract
from arXiv · showhide
What is the right supervisory signal to train visual representations? Current approaches in computer vision use category labels from datasets such as ImageNet to train ConvNets. However, in case of biological agents, visual representation learning does not require millions of semantic labels. We argue that biological agents use physical interactions with the world to learn visual representations unlike current vision systems which just use passive observations (images and videos downloaded from web). For example, babies push objects, poke them, put them in their mouth and throw them to learn representations. Towards this goal, we build one of the first systems on a Baxter platform that pushes, pokes, grasps and observes objects in a tabletop environment. It uses four different types of physical interactions to collect more than 130K datapoints, with each datapoint providing supervision to a shared ConvNet architecture allowing us to learn visual representations. We show the quality of learned representations by observing neuron activations and performing nearest neighbor retrieval on this learned representation. Quantitatively, we evaluate our learned ConvNet on image classification tasks and show improvements compared to learning without external data. Finally, on the task of instance retrieval, our network outperforms the ImageNet network on recall@1 by 3%
1 Introduction
The paper challenges passive visual learning by proposing physical interaction as supervision for visual representations. A Baxter robot collects interaction data across multiple tasks to train a shared ConvNet.
- Motivation: Category-label supervision dominates conventional visual feature learning, whereas biological agents can learn without supervision from other agents.Here, “unsupervised” allows supervision from other modalities or time.
- Motivation: Existing label-free approaches use passive observations, while the paper argues that visual learning requires physical exploration.The motivation is grounded in interactions such as pushing, grasping, and throwing objects.
- System: The system uses a Baxter robot with a gripper and tactile sensor to grasp, push, poke, and observe objects for visual representation learning.The paper reverses the usual pipeline by using robotic tasks to learn representations rather than only applying vision to tasks.
- System: A shared ConvNet combines task-specific outputs with shared early layers, allowing every physical interaction to provide a training datapoint.The learned ConvNet itself forms the visual representation.
- Data: 40,287 grasps, 5,472 pushes, 1,372 tactile observations, and 84,430 viewpoint pairs comprise the physical-exploration dataset.These task counts total more than 130K datapoints.
2 Related Work
The related work spans unsupervised representation learning and robotic interaction, but prior approaches primarily learn from passive data. This paper instead uses robotic tasks and sensor readings as supervision for visual representations.
- Scope: The paper connects unsupervised representation learning with the question of how agents interact with the world for learning.
- Unsupervised Learning: Prior unsupervised methods include generative models and discriminative auxiliary tasks using context, video viewpoints, ego-motion, or automatically obtained sensor supervision.
- Unsupervised Learning: These approaches observe passive data, whereas the paper treats physical interaction as a key requirement for training visual representations.
- Robotic Tasks: The robotic-task framework explores grasping, pushing, tactile sensing, and viewpoint-based identity vision using self-learning rather than human annotation.
3 Approach
The approach formulates four interaction-based tasks for learning visual representations, including planar grasp prediction from image patches. The grasp model predicts orientation-specific graspability rather than a single structured configuration.
- Task Formulation: The paper formulates planar grasping, planar pushing, poking, and identity vision across different object viewpoints as four learning tasks.
- Planar Grasps: The grasp dataset contains around 37K failed and 3K successful training interactions, with novel-object testing examples.
- Planar Grasps: Given an image patch, the grasp model outputs an 18-dimensional likelihood vector for graspability at angles from 0° through 170° in 10° increments.This patch formulation addresses multiple possible grasp locations and favors classification over structured-output regression.
3.2 Planar Push
The planar-push pipeline detects an object, executes a push between two 3D points, and records initial and final images. A siamese network learns to regress the action causing the observed transformation.
- Action Parameterization: The robot detects an object and parameterizes planar pushing with start and final 3D points, Xbegin and Xfinal.Because the push is planar, zbegin = zfinal; the points encode direction and velocity or force.
- Action Parameterization: The implemented push action uses five parameters: xstart, ystart, xfinal, yfinal, and zpushHeight.
- Data Collection: An off-the-shelf planner executes the push, after which the arm retracts and the final image Ifinal is recorded.The system collected 5K push actions on 70 objects.
- Learning from Push: A weight-sharing siamese network combines initial and final image features to regress the push action using mean squared error.The formulation captures the push magnitude, localization, and direction.
Push prediction formulation:
The robot collects poke data by pressing objects at sampled locations while recording tactile responses. These responses are modeled from images to predict a linear force-profile parametrization.
- 3.3 Poking: Tactile Sensing: A random point inside each object is sampled as the poke location before the robot pushes vertically into the table.
- 3.3 Poking: Tactile Sensing: The tactile graph’s profile provides cues about the material from which the object is made.
- 3.3 Poking: Tactile Sensing: The robot applies pressure until a limit threshold is exceeded, then pulls away and repeats the motion around 10 times per object.
- 3.3 Poking: Tactile Sensing: Tactile sensors attached to the robot’s finger continuously record pressure during the interaction.
- 3.3 Poking: Tactile Sensing: The tactile sensor increases electrical resistance monotonically with pressure, producing a corresponding voltage drop logged over time.
- 3.3 Poking: Tactile Sensing: Soft objects show more gradual force responses than harder objects during poking.
- 3.3 Poking: Tactile Sensing: Examples pair the poked object with force profiles observed by the tactile sensor.
- 3.3 Poking: Tactile Sensing: Given an object image, the ConvNet regresses the slope and intercept of a linear parametrization of the poke response.
Tactile prediction formulation:
The pose-invariance task uses image pairs from robot interactions to train an embedding in which views from the same interaction are closer than views from different interactions.
- 3.4 Identity Vision: Pose Invariance: Grasping interactions provide around five object images from multiple viewpoints, while planar pushes provide two images per interaction.
- 3.4 Identity Vision: Pose Invariance: The dataset contains around 42K positive image pairs and 42K negative pairs from different interactions.
- 3.4 Identity Vision: Pose Invariance: The embedding objective makes images from the same robot interaction close in fc7 feature space and images from different interactions distant.
3.5 Network Architecture
The architecture learns a common root representation through hierarchical feature sharing, while specialized task branches handle tasks of differing complexity and input requirements.
- 3.5 Network Architecture: Figure 8 summarizes a shared convolutional architecture serving four different tasks.
- 3.5 Network Architecture: The root network learns from every datapoint and connects to specialized task networks at different levels.
- 3.5 Network Architecture: Lower network levels predict simpler tasks, while more complex tasks use representations from deeper levels.
- 3.5 Network Architecture: The architecture supports tasks requiring either one image or two images as input.
- 3.5 Network Architecture: The root and cloned root streams share parameters and update them using gradients from every task.
- 3.5 Network Architecture: The root network follows an AlexNet-style layer scheme with convolutional, normalization, pooling, and fully connected layers.
Root network:
The root network feeds task-specific branches for grasping, pushing, tactile prediction, and identity embedding. Training accumulates root-network gradients across task batches before shared updates.
- Root network:: The grasp branch receives conv4 features and generates a classification loss backpropagated through the root network.
- Root network:: The grasp loss is backpropagated through the root chain, while root gradients wait for aggregation with gradients from other tasks.
- Root network:: The push branch processes paired conv3 outputs from the root network and its weight-sharing clone to produce an MSE regression loss.
- Root network:: The poke branch uses fc6 features to predict tactile responses with an MSE criterion.
- Root network:: Identity similarity is learned with a cosine embedding loss applied to fc7 features from paired root-network streams.
- Root network:: Training sequentially processes batches of 128 for each task and updates task branches during their respective backward passes.
4 Results
The learned representation captures visual structure from robot interactions and improves classification and instance retrieval relative to scratch, ImageNet, and unsupervised baselines, with mixed category-retrieval results.
- Classification: 35.4% accuracy on ImageNet Household is 10.4% higher than training from scratch.The evaluation uses a 25-class household-object dataset and fine-tunes the learned network.
- Classification: 69.3% accuracy on UW RGBD is about 22.5% higher than learning from a scratch network.The result uses leave-one-out methodology.
- Classification: 31.7% performance on Caltech-256 is 7.5% higher than learning from a scratch network.This result is reported alongside the UW RGBD classification evaluation.
- Baselines: A network trained on 150K rotation and viewpoint datapoints performs worse than the network trained with robot tasks.An auto-encoder trained on all robot data also performs worse than the proposed network.
- Image retrieval: 72% instance-level recall@1 exceeds ImageNet at 69% and randomNet at 6% on the RGBD retrieval evaluation.The comparison uses fc7 features and cosine distance.
- Image retrieval: 83% category-level recall@1 is slightly below ImageNet at 85%.The category-level retrieval result is reported separately from the instance-level comparison.
- Task ablation: Removing grasp data causes the largest classification-performance drop across all three datasets.The ablation excludes one of four tasks at a time, indicating grasp data may be the most important among the evaluated tasks.
5 Conclusion
The paper presents visual-representation learning from robot interactions with the physical world and validates the learned representation through visualization, classification, and retrieval tasks.
- Conclusion: The method uses over 130K physical-interaction datapoints to learn a meaningful visual representation.The authors describe this as a small step toward integrating robotics and vision.
- Conclusion: Visualization, classification, and retrieval evaluations are used to validate the physical-interaction learning hypothesis.The conclusion lists these evaluations as evidence for the learned representation.