Learning to See by Moving

Pulkit Agrawal; Joao Carreira; Jitendra Malik

Source-linked AI summary

Learning to See by Moving

Pulkit Agrawal, Joao Carreira, Jitendra Malik

arXiv:1505.01596v2May 7, 2015cs.CVcs.NEcs.RO

TL;DR

The paper asks whether visual features useful across tasks can be learned without millions of hand-labelled object-recognition images. It trains networks to predict camera transformations from image pairs using freely available egomotion, and reports favorable performance across scene recognition, object recognition, visual odometry, and keypoint matching. The authors also identify limited training data relative to model capacity as a scope limitation.

Problem
The paper asks whether useful visual representations for diverse visual tasks can be learned through supervision other than object-recognition labels.
Method
The method uses non-visual egomotion as self-supervision by training a siamese-like network to predict camera transformations between image pairs.
Results
Egomotion-based features compare favorably with class-label supervision across scene recognition, object recognition, visual odometry, and keypoint matching, including stronger performance with much less data on several tasks.
Takeaways & Limitations
Egomotion is a useful source of intrinsic supervision for visual feature learning in mobile agents.
Takeaways & Limitations
The study could not evaluate motion-based supervision across the full spectrum of training-set sizes because high-capacity models were trained on relatively little data.

Abstract

from arXiv · show

The dominant paradigm for feature learning in computer vision relies on training neural networks for the task of object recognition using millions of hand labelled images. Is it possible to learn useful features for a diverse set of visual tasks using any other form of supervision? In biology, living organisms developed the ability of visual perception for the purpose of moving and acting in the world. Drawing inspiration from this observation, in this work we investigate if the awareness of egomotion can be used as a supervisory signal for feature learning. As opposed to the knowledge of class labels, information about egomotion is freely available to mobile agents. We show that given the same number of training images, features learnt using egomotion as supervision compare favourably to features learnt using class-label as supervision on visual tasks of scene recognition, object recognition, visual odometry and keypoint matching.

1. Introduction

The paper asks whether useful visual representations can be learned without object-recognition labels and proposes egomotion as freely available supervision for mobile agents. It tests camera-transformation prediction on handwritten and real-world driving imagery.

Motivation: Object-recognition features trained on millions of labelled images transfer to several vision tasks, motivating alternative supervision for broader visual representation learning.The introduction contrasts object recognition with other uses of vision, including scene understanding and movement.
Motivation: Unsupervised learning methods had not yet produced useful features for complex real-world imagery or current applications.
Motivation: Useful representations are defined by supporting multiple visual tasks and learning new tasks from only a few labelled examples.
Approach: The method learns representations by correlating visual stimuli with egomotion, formulated as predicting camera transformation between consecutive image pairs.A mobile agent is treated as a camera moving through the world.
Approach: Camera-transformation prediction is intended to encourage features that identify visual elements shared between images, analogous to correspondence-oriented features such as SIFT.
Experiments: Experiments cover MNIST and real-world urban imagery from KITTI and San Francisco driving datasets, using image and odometry recordings from a moving car.The MNIST experiment compares against prior unsupervised feature learning when labels are available for only limited examples.

2. Related Work

Prior unsupervised approaches mainly reconstruct images, enforce transformation invariance, or model spatial transformations, while egomotion-based self-supervision uses non-visual motion information directly. The paper distinguishes its evaluated real-world method from closely related conceptual work and video methods that ignore camera motion.

Unsupervised feature learning: Image-reconstruction methods learn compact representations intended to retain enough information to reconstruct images.
Unsupervised feature learning: Other approaches learn transformation-invariant features from video or images, while some model spatial transformations without evaluating learned-feature quality.
Self-supervised alternatives: Because unsupervised learning had not solved complex real-world imagery, related work also explored intrinsic signals such as path traversability and direct driving prediction.
Egomotion supervision: The paper uses non-visual egomotion as self-supervision and predicts transformations between image pairs with a siamese-like network, unlike transforming auto-encoders that lacked feature-quality evaluation.
Video-based learning: Video-based methods impose similarity between temporally close frames without accounting for camera or object motion, whereas this approach uses camera-motion information.

3. A Simple Model of Motion-based Learning

The paper models motion-based feature learning as predicting egomotion from image pairs with a Siamese CNN, then evaluates the resulting representations through pretraining and finetuning on target tasks. On MNIST, egomotion pretraining outperforms several prior unsupervised approaches.

3. A Simple Model of Motion-based Learning: A Siamese CNN predicts camera transformation between two images, using shared-weight Base-CNN streams whose features feed a Top-CNN.The Top-CNN is discarded after pretraining, leaving a single Base-CNN for target-task feature extraction.
3. A Simple Model of Motion-based Learning: The experiments compare egomotion-based pretraining with class-label-based and slow-feature-based pretraining under a pretraining-finetuning evaluation paradigm.Pretraining optimizes a CNN for an auxiliary task, while finetuning modifies the pretrained weights for the target task.
3.3. Slow Feature Analysis (SFA) Baseline: SFA pretraining uses a contrastive loss that encourages nearby frames to have similar features and more distant frames to be separated by a margin.The distance measure D was chosen as L2 distance, with m as the margin and T as the temporal threshold.
3.4.1. Egomotion-Based Pretraining: On MNIST, egomotion is emulated with randomly transformed digit-image pairs, and CNNs are trained to classify the transformations between them.The transformations include translations and rotations, with separate soft-max losses for the X translation, Y translation, and Z-axis rotation.
3.4.3 Results: Egomotion-based pretraining outperforms convolutional deep belief networks, transformation-invariant feature learning, and SFA-based pretraining.The reported MNIST performance is evaluated using error rates with 100, 300, 1K, and 10K labelled examples.

4. Learning Visual Features From Egomotion in Natural Environments

Natural-environment feature learning uses image and odometry data from KITTI and SF, with camera transformations discretized into classification targets. The resulting filters are mainly color detectors, with edge-detector prevalence differing between the datasets.

4.1. KITTI Dataset: KITTI supplies 23,201 frames from 11 car trips, with 20,501 training images from nine sequences and two validation sequences.The data were recorded while a car moved through urban landscapes.
4.1. KITTI Dataset: KITTI camera transformations use translation along the Z/X axes and rotation about the Y axis, each discretized into 20 uniformly spaced bins.Image pairs were selected from frames at most ±7 frames apart to maintain reasonable overlap.
4.3. Network Architecture: The natural-environment SCNN is trained on randomly cropped 227 × 227 regions from larger images, using an AlexNet-like BCNN and a multilayer TCNN.The BCNN follows C96-P-C256-P-C384-C384-C256-P, while the TCNN uses C256-C128-F500-D-Op.
4.2. SF Dataset: SF provides approximately 136K camera-transformation-labelled image pairs from 17,357 unique Google StreetView images, including approximately 130K training pairs and 6K validation pairs.Unlike KITTI, SF contains significant transformations across all six dimensions.
4.3. Network Architecture: Most layer-1 filters in KITTI-Net and SF-Net detect colors, while some detect edges because color supports correspondence between nearby frames.SF-Net has a higher fraction of edge detectors, consistent with more images containing structured objects such as buildings and cars.

5. Evaluating Motion-based Learning

The evaluation compares egomotion-, class-label-, and SFA-based features across scene recognition, object recognition, keypoint matching, and visual odometry under limited supervision. Egomotion-based features generally perform competitively or better, including across tasks with constrained target-task labels.

Evaluation setup: Evaluations covered scene recognition, object recognition, intra-class keypoint matching, and visual odometry, often using only 1–20 labelled examples per class for finetuning.The comparisons included egomotion-, class-label-, and SFA-based supervision.
Scene Recognition: KITTI-Net was comparable to AlexNet-20K for SUN scene recognition using the same 20K-image pretraining budget.KITTI-Net also outperformed GIST but was outperformed by Dense SIFT with spatial pyramid matching.
Discussion: KITTI-Net’s scene-recognition performance was notable because it used limited, visually narrow KITTI data yet competed with AlexNet-20K trained on more diverse imagery.The authors expect more diverse egomotion-training data to improve performance further.
Object Recognition: KITTI-Net clearly outperformed KITTI-SFA-Net and AlexNet-Scratch for ILSVRC-12 object recognition, with larger gains when fewer target-task examples were available.Networks were finetuned with 1, 5, 10, or 20 examples per class.
Intra-Class Keypoint Matching: KITTI-Net outperformed AlexNet-20K and AlexNet-100K and was inferior only to AlexNet-1M for intra-class keypoint matching.It also significantly outperformed KITTI-SFA-Net, while SIFT performed as well as the best AlexNet-1M conv-4 features.
Visual Odometry: KITTI-Net was either superior or comparable to AlexNet-1M for visual odometry, although SF-Net outperformed it on some metrics when evaluated on the SF dataset.The authors interpret these results as evidence that egomotion-based feature learning is superior to class-label-based learning for visual odometry.

6. Discussion

Egomotion provides intrinsic supervision for visual feature learning and performs competitively across several tasks, while class-label supervision is not optimal for every visual task. The study also identifies limited data and the pretraining setup as important boundaries for future work.

6. Discussion: Egomotion-based features perform almost as well as class-label-based features for scene recognition with the same pretraining-image budget.
6. Discussion: Egomotion-based features outperform CNN features trained with 100 times more data for visual odometry and 10 times more data for intra-class keypoint matching.
6. Discussion: The results suggest that class-label supervision is not optimal for all visual tasks, motivating investigation of task-specific pretraining.
6. Discussion: High-capacity models were trained and evaluated with relatively little data, including approximately 20K unique KITTI images, and no egomotion dataset as large as ImageNet was available.
6. Discussion: The experiments pretrain models on egomotion and then finetune them, leaving online learning with continuous intrinsic and occasional extrinsic supervision as an open setting.

A. Keypoint Matching Score

For keypoint matching, images of same-class object instances are cropped, padded, and resized before feature-based correspondence is evaluated.

A. Keypoint Matching Score: The procedure evaluates pairs of images containing two instances of the same object class.
A. Keypoint Matching Score: Each image is preprocessed by cropping its groundtruth bounding box.
A. Keypoint Matching Score: The cropped images are padded by 30 pixels along each dimension.
A. Keypoint Matching Score: Each image is resized so its smallest side is 227 pixels while preserving aspect ratio.

A.1. Keypoint Matching using CNN

CNN keypoint matching assigns feature vectors to spatial grid locations, selects the closest matching feature, and computes normalized localization error averaged across pairs and classes.

A.1. Keypoint Matching using CNN: A CNN layer produces an M-dimensional feature vector at each of the I × J spatial positions in its feature map.
A.1. Keypoint Matching using CNN: Each keypoint is assigned to the feature-map grid position whose receptive-field center is closest in image coordinates.
A.1. Keypoint Matching using CNN: The matching location in the second image is selected by minimizing feature distance between the first-image keypoint vector and candidate grid-location vectors.
A.1. Keypoint Matching using CNN: The selected grid location is transformed to image coordinates using its receptive-field center and compared with the groundtruth keypoint location.
A.1. Keypoint Matching using CNN: The kth-keypoint error is normalized by the second image's diagonal length, D, to account for different image sizes.
A.1. Keypoint Matching using CNN: Average matching error is computed across same-class object-instance pairs and then averaged across all 20 classes.

A.2. Keypoint Matching using SIFT

SIFT matching uses a fixed square window and stride, while CNN features use layer-dependent strides; both are scored with the same matching-error procedure.

A.2. Keypoint Matching using SIFT: SIFT features are extracted with a 72-pixel square window and an 8-pixel stride.
A.2. Keypoint Matching using SIFT: CNN feature strides are 8 pixels for conv-2 and 16 pixels for conv-3, conv-4, and conv-5.
A.2. Keypoint Matching using SIFT: SIFT matching error is calculated in the same way as for CNN features.

A.3. Effect of Viewpoint on Keypoint Matching

The section evaluates keypoint-matching accuracy across increasing viewpoint distances, motivated by the expectation that larger transformations make matching harder. Viewpoint distance is computed from rotation matrices and summarized in cumulative distance bins.

Larger viewpoint distances are expected to make matching instances of the same object more difficult.
Viewpoint distance is measured with a Riemannian metric on the manifold of rotation matrices.The metric is ||log(R_iR_j^T)||_F, using the matrix logarithm and Frobenius norm.
10 uniform viewpoint-distance bins of 18° each organize the matching analysis.
Keypoint-matching error increases as viewpoint distance becomes larger.The error for bin k includes all instance pairs with viewpoint distance ≤ k×18°.