Source-linked AI summary
Pairwise Decomposition of Image Sequences for Active Multi-View Recognition
Edward Johns, Stefan Leutenegger, Andrew J. Davison
TL;DR
The paper addresses CNN-based multi-view recognition when camera trajectories and sequence lengths are arbitrary. It decomposes sequences into image pairs, adds discriminative next-view prediction and trajectory optimisation, and reports state-of-the-art results on ModelNet across guided and unguided recognition.
Problem
Existing multi-view recognition methods rely heavily on hand-crafted geometric models, while fixed-length CNN inputs cannot cover arbitrary camera paths and sequence lengths.
Method
The method classifies every image pair independently, weights pair contributions for recognition, trains a CNN for direct next-view prediction, and optimises trajectories using all observable images.
Results
The method achieves state-of-the-art results on ModelNet for multi-view recognition, next-view prediction, and trajectory optimisation, using greyscale, depth, or combined images.
Takeaways & Limitations
Pairwise decomposition enables recognition over unconstrained camera trajectories while extending naturally to active recognition and trajectory planning.
Takeaways & Limitations
The method is evaluated on ModelNet CAD meshes without real-world experiments, and assumes canonical alignment, known camera pose, and no occlusion or clutter.
Abstract
from arXiv · showhide
A multi-view image sequence provides a much richer capacity for object recognition than from a single image. However, most existing solutions to multi-view recognition typically adopt hand-crafted, model-based geometric methods, which do not readily embrace recent trends in deep learning. We propose to bring Convolutional Neural Networks to generic multi-view recognition, by decomposing an image sequence into a set of image pairs, classifying each pair independently, and then learning an object classifier by weighting the contribution of each pair. This allows for recognition over arbitrary camera trajectories, without requiring explicit training over the potentially infinite number of camera paths and lengths. Building these pairwise relationships then naturally extends to the next-best-view problem in an active recognition framework. To achieve this, we train a second Convolutional Neural Network to map directly from an observed image to next viewpoint. Finally, we incorporate this into a trajectory optimisation task, whereby the best recognition confidence is sought for a given trajectory length. We present state-of-the-art results in both guided and unguided multi-view recognition on the ModelNet dataset, and show how our method can be used with depth images, greyscale images, or both.
1. Introduction
The paper targets CNN-based recognition over arbitrary multi-view camera trajectories by decomposing sequences into image pairs, then extends this representation to next-view prediction and trajectory optimisation.
- Motivation: Multi-view recognition is more realistic than single-image recognition for active applications such as robot object rotation and semantic room mapping.Efficient camera control matters when time or power constraints apply.
- Approach: The method decomposes an image sequence into image pairs, enabling recognition and trajectory planning without direct training over every possible camera path.This makes pairwise training tractable for unconstrained trajectories.
- Problem: Fixed-length sequence CNNs cannot readily handle arbitrary paths and lengths, while concatenating all images would require training over potentially infinitely many inputs.The proposed decomposition relaxes the joint model over the full sequence.
- Active recognition: A second CNN predicts the next viewpoint directly from an observed image, providing a discriminative alternative for active recognition.The contribution is framed as direct Next-Best-View prediction.
- Trajectory optimisation: A third contribution optimises trajectories by considering the impact of all observable images along the sequence rather than only selected next views.The paper reports state-of-the-art results for these contributions on ModelNet.
2. Related Work
Prior multi-view recognition uses view-based features, sequence CNNs, or explicit 3D shape models, but existing sequence methods assume fixed lengths and 3D alternatives have lower reported performance.
- View-Based Multi-View Recognition: View-based methods build viewpoint-tolerant descriptors or track features shared across multiple views.These approaches rely on handcrafted features, correspondences, or learned relative displacements.
- View-Based Multi-View Recognition: Sequence CNNs use max pooling across viewpoints or panorama rows, but assume fixed-length sequences during training and testing.That assumption makes them unsuitable for generalised multi-view recognition.
- Shape-Based Multi-View Recognition: Shape-based methods represent objects through reconstructed or CAD-derived 3D geometry and match depth images, reconstructions, or completed shapes.Descriptors include local surface properties, spherical harmonics, and 3D invariant features.
- Shape-Based Multi-View Recognition: CNNs applied to 3D shapes use occupancy grids with generative or discriminative networks.These methods provide voxel-based alternatives to 2D view-based recognition.
- Shape-Based Multi-View Recognition: Reportedly, 2D view-based CNN methods outperform 3D voxel-based counterparts, although the source of this advantage remains unclear.Possible explanations include more abundant 2D pre-training data or more efficient 2D representations.
- Active Recognition: Active-recognition methods generally generate predicted appearances from unvisited viewpoints and select views using entropy reduction or feature discovery.Prior work includes SIFT-based selection, guided mapping, and robot navigation.
- Active Recognition: Many active-recognition approaches ignore images acquired while moving toward the selected view, while POMDPs optimise trajectories for particular tasks.The paper positions trajectory-wide image contributions as an open consideration for recognition.
3. Multi-View Object Recognition
The recognition system renders full-sphere views, forms pairwise inputs with relative pose, classifies each pair using CNN-1, and combines weighted pair predictions for sequence recognition.
- Dataset and viewpoint representation: Viewpoints are discretised over the full viewing sphere, allowing camera rotations beyond gravity-axis constraints and supporting arbitrary trajectories.The camera pose uses spherical coordinates, with θ and φ changing by one 30° step or remaining constant.
- Dataset and viewpoint representation: The experimental setup assumes canonical object alignment, known camera pose, CAD meshes without occlusion or clutter, and a preceding detection and segmentation stage in practice.The authors note that model rotation, tracking, reconstruction, robot kinematics, or external sensors could relax some assumptions.
- Pairwise decomposition: A sequence of M views is decomposed into N = M(M−1)/2 view pairs, with each new view paired with every earlier view.The final recognition score aggregates information from all pairwise observations.
- Pairwise decomposition: Each pair contains two images and their relative camera pose, and each pair is classified independently before weighted aggregation.The pairwise classifiers act as weak classifiers whose weighted average forms the final class-score distribution.
- CNN-1: CNN-1 predicts the object class from a view pair using parallel image-processing streams with shared weights.The architecture is designed for greyscale images, depth images, or both modalities.
- CNN-1: The Siamese-inspired design reduces learned parameters and avoids the larger capacity of concatenating both images into one input vector.The architecture concatenates final image features with a one-hot relative-pose representation and can combine modalities.
- Confidence weighting: Relative pose conditions confidence weighting because viewpoint separation can affect how much object coverage a pair provides.Cross entropy is averaged by relative pose to learn weights that emphasize more informative pairs.
4. Active Object Recognition
Active recognition predicts informative camera movements from observed views and extends next-best-view selection into trajectory optimisation. The method uses CNNs to estimate pairwise recognition costs and chooses trajectories that accumulate high-confidence evidence across views.
- Next-best-view prediction: CNN-2 directly predicts the next-best viewpoint from an input image for active recognition.It replaces generative view synthesis with a discriminative CNN-based prediction.
- Trajectory optimisation: Unlike sequential next-best-view traversal, trajectory optimisation accounts for images acquired while moving and aims to benefit from every observable view pair.The next-best-view solution is therefore described as not globally optimal because it ignores intermediate moving-camera observations.
- Trajectory optimisation: The trajectory heuristic scores unobserved viewpoints by summing predicted cross entropies over pairs with all currently observed views.These costs are updated whenever a new view is added to the sequence.
- Trajectory optimisation: CNN-3 predicts cross entropies for candidate viewpoints by regressing from a single image to values over the viewing sphere.Its training targets come from CNN-1 classification cross entropies for paired views, and it minimizes L2 distance to those targets.
- Trajectory optimisation: At each step, the method searches candidate continuations and selects the next view whose best available trajectory has the highest score.The candidates are the eight viewpoints adjacent to the current camera position, and trajectories are generated for a fixed final length.
5. Experiments
Experiments on ModelNet evaluate pairwise multi-view recognition across modalities, sequence types, and active trajectories. The method achieves strong recognition on constrained and arbitrary trajectories, with trajectory optimisation improving performance over random selection.
- Datasets: ModelNet10 contains 4,905 objects across 10 categories, while ModelNet40 contains 12,311 objects across 40 categories for training and testing.The dataset consists of 3D CAD meshes of everyday objects; real-world experiments were not possible because no comparable large-scale multi-view dataset was available.
- Implementation: Training renders every model from discretised viewing-sphere viewpoints to form image pairs, producing 512-by-512 greyscale and depth images.Greyscale rendering uses Phong shading, with ImageNet pre-training described for the greyscale network.
- Multi-View Recognition: Using all view pairs and learned pair weights improves recognition over equal weighting or selecting only the best pairs in six-view random trajectories.All-pair selection uses M(M−1)/2 pairs, whereas best-pair selection uses the top M pairs by λ, reducing classification growth from combinatorial to linear in sequence length.
- Multi-View Recognition: The method outperforms ShapeNets and MVCNN on 30° gravity-constrained trajectories, while combining greyscale and depth yields a small boost over either modality alone.MVCNN performs second-best for 360° sequences but degrades sharply on shorter sequences; the proposed method achieves comparable performance with half as many views.
- Multi-View Recognition: On arbitrary trajectories, the method achieves state-of-the-art recognition, and straight paths outperform random paths because random walks may revisit viewpoints or remain locally confined.Results are slightly lower than for the 30° constrained setting, attributed to the suitability of those viewpoints for common household objects in canonical orientation.
- Active Recognition: On ModelNet40, trajectory optimisation maintains an advantage over random viewpoint selection across a range of sequence lengths, with optimised trajectories often passing over or beneath objects.The active-recognition evaluation compares active and random trajectories using adjacent viewpoints; example greyscale observations are visualised for ModelNet10.
6. Conclusions
The paper presents pairwise multi-view recognition for unconstrained camera trajectories and extends it to next-best-view prediction and trajectory optimisation. Experiments report state-of-the-art recognition on ModelNet across greyscale, depth, and combined modalities.
- The method recognises objects over unconstrained camera trajectories using greyscale images, depth images, or both modalities.
- Decomposing image sequences into view pairs makes training tractable for any trajectory over an object's viewing sphere.
- The pairwise method extends to next-best-view prediction through end-to-end discriminative learning.
- Trajectory optimisation seeks the best camera path for recognition over a given sequence length.
- The approach outperforms voxel-based generative ShapeNets and Multi-View CNN, achieving state-of-the-art recognition on ModelNet.