Source-linked AI summary
Learning image representations tied to ego-motion
Dinesh Jayaraman, Kristen Grauman
TL;DR
Visual learning methods often ignore the physical source of images, despite the importance of ego-motion for understanding changing visual scenes. The paper uses proprioceptive motor signals from egocentric video to regularize CNN features toward ego-motion equivariance, and reports consistent gains across tasks and datasets, including cross-domain scene recognition.
Problem
Existing visual learning methods are disconnected from the physical source of images, while proper perceptual development requires self-generated movement alongside visual feedback.
Method
The method couples synchronized ego-motor signals and video in an unsupervised CNN objective that learns individual-image features with predictable responses to discrete ego-motion patterns.
Results
Across three datasets and multiple tasks, the approach consistently outperforms related feature-learning methods and improves recognition when vehicle-video features transfer to the disjoint SUN image domain.
Takeaways & Limitations
The results support embodied feature learning as an alternative to treating images as an isolated “bag of images.”
Takeaways & Limitations
Appearance changes depend not only on ego-motion but also on scene depth and dynamic objects, and the supervised recognition or test data need not include sensor readings.
Abstract
from arXiv · showhide
Understanding how images of objects and scenes behave in response to specific ego-motions is a crucial aspect of proper visual development, yet existing visual learning methods are conspicuously disconnected from the physical source of their images. We propose to exploit proprioceptive motor signals to provide unsupervised regularization in convolutional neural networks to learn visual representations from egocentric video. Specifically, we enforce that our learned features exhibit equivariance i.e. they respond predictably to transformations associated with distinct ego-motions. With three datasets, we show that our unsupervised feature learning approach significantly outperforms previous approaches on visual recognition and next-best-view prediction tasks. In the most challenging test, we show that features learned from video captured on an autonomous driving platform improve large-scale scene recognition in static images from a disjoint domain.
1. Introduction
The paper frames visual learning as an embodied process shaped by self-generated movement, arguing that ego-motion signals should be learned jointly with pixels to produce equivariant representations. It introduces an unsupervised CNN approach and reports improvements across datasets, including transfer from vehicle video to static scene recognition.
- Self-produced movement and visual feedback jointly support proper perceptual development.
- “Bags of images” treat snapshots as independent and ignore the physical source of visual experience.Video alone only partially addresses this problem when accompanying motor signals are unused.
- The proposed embodied approach learns representations from parallel ego-motion and pixel signals.Its goal is to preserve the connection between how the observer moves and how surroundings change.
- The method trains a CNN to produce features that change predictably with discrete ego-motion patterns, using synchronized sensor readings and image sequences without semantic labels.Motion patterns are mined from pose differences between temporally close video frames.
- Unlike prior feature learning focused mainly on invariance or passive scene dynamics, this work grounds feature learning in physical activity and 3D ego-motion.The paper positions this as a novel coupling of ego-motor signals and video.
- Across three public datasets, the approach consistently outperforms related feature-learning methods and transfers vehicle-video features to recognition in a disjoint image domain.The authors describe this as evidence for moving beyond a “bag of images” mindset toward embodied feature learning.
2. Related work
Prior work studies invariance, temporal coherence, image-transformation equivariance, disentangled factors, transformation representations, and egocentric cues. The paper distinguishes its approach by learning individual-image features whose responses reflect natural 3D observer motion rather than relying only on 2D transformations or passive visual dynamics.
- Invariant features: Invariant representations keep outputs unchanged under transformations, including small shifts and rotations targeted by SIFT, HOG, pooling, and convolution.
- Temporal coherence: Temporal-coherence methods require learned features to vary slowly across adjacent video frames and support tasks such as recognition, dimensionality reduction, and metric learning.
- Equivariant representations: Earlier equivariant feature-learning methods craft equivariance for in-plane 2D translation and rotation, whereas this paper learns responses to complex natural 3D ego-motion.
- Disentangled representations: Disentangled latent-factor methods separate properties such as pose and illumination, often appending pose parameters to otherwise invariant feature spaces.
- Equivariance measurement: Some work measures equivariance in existing descriptors, while this approach learns a feature space with the desired equivariance.
- Learning transformations: Transformation-learning models encode relations between image pairs or tuples, while this method learns representations for individual images with predictable behavior under transformations.
- Egocentric vision: Egocentric-vision methods use motion cues for tasks such as foreground-background separation and gaze inference, but do not learn features from motor signals and pixels together.
3. Approach
The approach learns image representations whose responses change predictably with ego-motion, using unlabeled image pairs annotated by discretized motion patterns. It jointly learns equivariant feature maps and recognition-compatible representations while accounting for scene-dependent limitations and compositional motions.
- Mining discrete ego-motion patterns: Training uses unlabeled image pairs with synchronized ego-poses, replacing pose differences with discrete motion-pattern IDs discovered from temporally close video frames.The authors apply k-means to pose-difference vectors, typically from frames ⪅1 second apart; KITTI clusters include turning left, turning right, and going forward.
- Ego-motion equivariance: The method learns a mapping zθ from individual images to a D-dimensional feature space that responds predictably to ego-motion transformations.Equivariance is represented by a D × D matrix Mg mapping features before a motion to features after it.
- Ego-motion equivariance: Appearance changes are not determined by ego-motion alone because they also depend on scene depth and dynamic objects, motivating training on temporally close pairs.The experiments nevertheless indicate effective representations with dynamic objects, while close temporal pairing avoids some changing-occlusion and newly appearing-object pitfalls.
- Ego-motion equivariance: Linear equivariance maps can be composed, allowing equivariance to novel motions formed from sequences of trained atomic motions.For a diagonal motion d = r ◦u, the corresponding map is Md = MrMu even when d was not among the training patterns.
- Equivariant feature learning objective: The learning objective jointly optimizes feature parameters and motion-specific equivariance maps, while contrastive terms prevent trivial all-zero features and distinguish different motions.The naive equivariance objective admits zero-feature solutions, so motion-specific negatives are included with a margin-based contrastive loss.
- Regularizing recognition: The approach can jointly regularize supervised recognition through a softmax loss, and the resulting features can be used for recognition without sensor data in labeled or test images.Neither supervised training data nor recognition test data needs associated sensor readings.
4. Experiments
The experiments evaluate ego-motion-equivariant feature learning across equivariance, recognition, and next-best-view tasks on three datasets. EQUIV consistently outperforms related feature-learning methods, including in cross-domain KITTI-to-SUN recognition with few labels.
- Experimental setup: The study compares CLSNET, TEMPORAL, DRLIM, EQUIV, and EQUIV+DRLIM across equivariance, recognition, and next-best-view selection tasks.The datasets pair unsupervised data with supervised recognition data, using NORB, KITTI, and SUN.
- Experimental setup: NORB contains 24,300 96×96-pixel images of 25 toys with camera elevation and azimuth as ego-pose variables.The experiment uses atomic elevation and azimuth motion patterns, with a 67%-33% train-validation split.
- Experimental setup: KITTI supplies car videos with registered GPS/IMU streams across campus, city, residential, and road areas, using yaw and forward position as ego-pose variables.Six ego-motion clusters are discovered from frame pairs no more than one second apart.
- Equivariance measurement: 0.0304 and 0.0394 average ρg error for atomic and composite NORB motions show that EQUIV learns near-complete equivariance, versus 0.3751 and 0.4532 for DRLIM.Composite motions test generalization beyond the motion patterns used during training.
- Equivariance measurement: EQUIV features are most equivariant after recognition training, although all methods have lower error for atomic than composite motions.The result is attributed to greater equivariance for smaller motions.
- Recognition results: On all three recognition datasets, EQUIV significantly improves classification accuracy over CLSNET and the closest previous unsupervised feature-learning methods.All unsupervised methods improve over CLSNET, while DRLIM and TEMPORAL are weaker than EQUIV.
- Recognition results: KITTI-to-SUN recognition reaches 1.58% accuracy with six labeled examples per class, approximately 30% above TEMPORAL and over six times chance.KITTI video differs from SUN’s web-image domain, and the KITTI setting includes noisy poses, dynamic scenes, occlusions, and depth variation.
- Recognition results: With N=20 samples per class on KITTI-SUN, EQUIV scores 3.66+/-0.08% accuracy versus 1.66+/-0.18 for CLSNET.The authors report that EQUIV’s advantage is preserved with larger labeled training sets.
5. Conclusion
The paper frames visual recognition as an embodied process rather than learning from disembodied image collections. It presents ego-motion-equivariant features and reports benefits across multiple datasets and downstream tasks.
- 5. Conclusion: Image collections that discard the physical context of visual experience limit feature learning compared with embodied visual data.The conclusion contrasts large-scale “bags of images” with representations informed by how images arise through physical experience.
- 5. Conclusion: The paper presents an embodied feature-learning approach that generates features equivariant to ego-motion.Results across multiple datasets and tasks show successful learning of equivariant features that benefit downstream tasks.
6. Supplementary details
The supplementary material details the datasets, architectures, optimization choices, equivariance measurement, and analyses of slowness, next-best-view selection, and qualitative feature behavior. Results show strong equivariance, generalization to composite motions, and near-perfect slowness prediction early in training.
- Datasets: KITTI and SUN differ substantially: KITTI contains fixed-view grayscale street scenes from four location classes, whereas SUN contains 397 diverse indoor and outdoor categories.KITTI images face the road with fixed field of view and pitch; SUN images are internet-downloaded and mostly unrelated to roads.
- Network architecture: The KITTI network uses three convolutional layers followed by a fully connected layer producing D = 64-dimensional features.The architecture follows a CIFAR-10-style cuda-convnet design with ReLU and pooling operations.
- Optimization and hyperparameters: Hyperparameters were selected by greedy cross-validation, with learning rates chosen for CLSNET and regularizer weights selected from logarithmic grids for the other methods.The contrastive-loss margin δ was set to 1.0 for DRLIM, TEMPORAL, and EQUIV; EQUIV+DRLIM used margins 1.0 and 0.1.
- Equivariance measurement: ρg measures normalized equivariance error, with ρg = 0 indicating perfect equivariance; maps for baselines were estimated by least squares on held-out validation data.Equivariance maps and errors were computed on disjoint validation subsets, and the output feature dimension was D = 100 for this evaluation.
- Equivariance results: Equivariance errors are lower for smaller atomic motions, while composite-motion errors remain comparable to larger atomic-motion errors, indicating generalization to novel motions.The 5° “up” motion has lower ρg than the 20° “right” motion, and composite motions do not systematically increase error beyond the larger atomic motion.
- Slowness analysis: Slowness AUROCs were already near 0.9 before optimization and reached nearly 1.0 on training and testing data within about 4000 iterations.This early saturation supports the claim that slowness provides a weak prior for the compared methods.