Source-linked AI summary
Orientation-boosted Voxel Nets for 3D Object Recognition
Nima Sedaghat, Mohammadreza Zolfaghari, Ehsan Amiri, Thomas Brox
TL;DR
3D classifiers may learn only partial representations of pose, although orientation matters for recognizing rotated objects. The paper adds class-specific orientation prediction as an auxiliary task to a voxel-based network and evaluates it across multiple 3D datasets and detection settings. Orientation-boosted training improves classification consistently, reaches state-of-the-art results on most datasets, and can substantially accelerate detection when its orientation output guides box rotation.
Problem
Direct class-label training may produce representations that only partially generalize, despite the need to handle object orientation in 3D recognition.
Method
The paper adds class-specific azimuth orientation classification as a parallel auxiliary task to a VoxNet-based 3D voxel classifier.
Results
Orientation-boosted training improves classification consistently across datasets, establishes state-of-the-art results on most of them, and improves detection runtime when orientation predicts box orientation.
Takeaways & Limitations
Learning orientation alongside object class supports orientation-related invariance in 3D classification and avoids exhaustive rotation search in the reported detection setting.
Abstract
from arXiv · showhide
Recent work has shown good recognition results in 3D object recognition using 3D convolutional networks. In this paper, we show that the object orientation plays an important role in 3D recognition. More specifically, we argue that objects induce different features in the network under rotation. Thus, we approach the category-level classification task as a multi-task problem, in which the network is trained to predict the pose of the object in addition to the class label as a parallel task. We show that this yields significant improvements in the classification results. We test our suggested architecture on several datasets representing various 3D data sources: LiDAR data, CAD models, and RGB-D images. We report state-of-the-art results on classification as well as significant improvements in precision and speed over the baseline on 3D detection.
1 Introduction
The paper argues that object orientation is an important, difficult component of 3D recognition and can be learned as an auxiliary task to improve classification. It introduces an orientation-boosted network evaluated across varied 3D data sources and applied to faster 3D detection.
- Motivation: 3D recognition increasingly uses learned features from point clouds acquired by range sensors, depth cameras, structure from motion, and SLAM.The paper frames 3D convolutional networks as requiring only small conceptual changes from 2D recognition.
- Motivation: Auxiliary orientation learning addresses partial generalization when networks must learn underlying concepts such as object pose.Directly training only on class labels may not provide a signal that pose-related concepts are important.
- Core idea: Forcing correct orientation prediction during training significantly increases category-level classification accuracy.The classification task requires some orientation-invariant representation, which is nontrivial in 3D.
- Evaluation: The architecture is evaluated on four datasets spanning laser range scanners, RGB-D images, and CAD models, improving results in all experiments.The study compares the baseline without orientation information against the orientation-boosted version and reports state-of-the-art results in most experiments.
- Detection: In 3D detection, the orientation output determines the box orientation and largely reduces detector runtime.This extends the classifier beyond category-level recognition to a simple sliding-box detection setting.
3 Method
The method augments a VoxNet-style voxel classifier with class-specific orientation prediction as a parallel task. Orientation is discretized to accommodate object symmetries and rough alignment, and the two task losses are combined during training; test-time voting can aggregate rotated inputs.
- Architecture: The baseline uses a 3D voxel grid, two 3D convolutional layers, and two fully connected layers, with a deeper four-convolutional-layer variant also tested.The architecture is based on VoxNet to enable direct comparison with the baseline.
- Multi-task learning: ORION adds orientation estimation as an auxiliary parallel task and treats orientation as azimuth rotation around the z-axis.The paper uses “orientation” specifically for this rotational component.
- Orientation labels: Orientation estimation is formulated as classification rather than continuous regression, allowing rough alignment instead of strict orientation annotations.The network predicts from a product label space of object classes and orientation classes.
- Orientation labels: Orientation labels remain class-specific because sharing labels across classes would encourage features unrelated to the desired classification improvement.The paper gives a table and car at the same angle as an example that need not share useful information.
- Training objective: The classification and orientation cross-entropy losses are summed with equal weights, using γ = 0.5 in the classification experiments.A higher orientation-loss weight is used in one detection experiment to improve orientation accuracy.
- Orientation labels: Different object classes receive different numbers of orientation labels to account for symmetries and rotationally neutral objects.Symmetric objects can use a single orientation node, while label counts are otherwise assigned per class.
- Inference: At test time, voting feeds multiple rotated versions of an object to the network and aggregates class scores into a final consensus.S_k is the score assigned to class k, and x_r denotes the test input with rotation index r.
4 Datasets
The experiments use four datasets spanning LiDAR point clouds, RGB-D images, and synthetic CAD models, with varied alignment and annotation conditions.
- Four datasets represent different 3D data sources: Sydney and KITTI use LiDAR point clouds, NYUv2 uses RGB-D images, and ModelNet uses synthetic CAD models.Three datasets are illustrated in Figure 3; KITTI is similar to Sydney and is omitted from the figure.
- Sydney contains 631 incomplete LiDAR-scanned objects across 26 categories, including several challenging vehicle classes.Objects are observed from a single viewpoint, making the point clouds difficult to classify.
- NYUv2 contains 2,808 RGB-D images covering 10 object classes, represented as 32x32x32 voxel grids with up to 12 rotations.Because NYUv2 lacks orientation annotations, the experiments use annotations provided by SUN-RGBD.
- ModelNet10 provides uniformly aligned CAD models, whereas ModelNet40 lacks alignments and receives manual and unsupervised orientation alignment.ModelNet meshes are converted to voxel grids, and the ModelNet40 alignment procedure is described in the supplementary material.
- KITTI supplies paired camera images and Velodyne point clouds with 2D and 3D box annotations, but is used only for detection.Camera calibration converts image annotations into Velodyne scanner coordinates.
5 Experiments and Results
Across classification and detection experiments, orientation supervision improves recognition, while using predicted orientation substantially accelerates 3D detection.
- 5.1 Classification: The auxiliary orientation task improves classification accuracy on all three evaluated datasets and yields state-of-the-art results.The Sydney metric is weighted average F1; the other datasets use average accuracy.
- 5.1 Classification: The proposed classifier is much shallower and smaller than prior state-of-the-art methods, using 2 versus 43 convolutional layers and 1M versus 18M parameters.
- 5.1.1 Non-aligned Dataset: Orientation information boosts ModelNet40 classification even with unsupervised alignment, while manual labels provide no significant additional improvement.Batch normalization and two additional convolutional layers further improve results.
- 5.2 Detection: An 18× runtime improvement results when the detector uses the orientation output to directly predict object orientation instead of exhaustively searching 18 rotation steps.Using the architecture as a binary classifier also significantly raises the precision-recall curve and mean average precision.
- 5.2 Detection: The detector uses 3D point clouds alone rather than KITTI RGB images and limits scale and aspect-ratio search using training-set car-size statistics.
6 Analysis
Analysis indicates that orientation-boosted training changes how features are distributed across filters for different object orientations.
- ORION spreads feature contributions across different filters for different orientations, unlike the baseline’s dominant filters across all rotations.The comparison initializes ORION from a stabilized baseline and continues training at a low learning rate.
7 Conclusions
The paper finds that auxiliary orientation learning supports 3D object classification, improving results consistently across datasets and achieving state-of-the-art performance on most of them. In sliding-box 3D detection, the orientation output avoids exhaustive rotation search.
- Auxiliary orientation prediction supports learning orientation-invariant representations for 3D object classification.The network is trained to produce object orientation alongside the class label.
- Better classification results were achieved consistently across all evaluated datasets.The paper reports state-of-the-art results on most datasets.
- The orientation output avoids exhaustive search over object rotations in simple sliding-box 3D detection.Here, the orientation prediction is used directly in the detection process.
8 Auto-Alignment of the Modelnet40 dataset
The ModelNet40 preparation converts meshes to point clouds, removes hidden surfaces, reconstructs lighter models, and automatically aligns objects class by class. The orientation-class count is selected separately for each category.
- Auto-Alignment: ModelNet40 contains more than 12000 non-aligned objects in 40 classes, which were automatically aligned class by class.The objects were aligned using the method of Sedaghat and Brox.
- Mesh to Point-Cloud Conversion: Mesh grids were converted to point clouds by assigning uniformly distributed points to object faces.The conversion provides the point-cloud representation required by the auto-alignment method.
- Mesh to Point-Cloud Conversion: Hidden mesh faces were removed using ambient occlusion because the HOH descriptor mainly relies on exterior surfaces.Jacobson’s implementation of ambient occlusion was used for this preprocessing step.
- Mesh to Point-Cloud Conversion: Around 50000 points per object were generated with roughly uniform density across faces before SSD surface reconstruction produced lighter models.The density choice was intended to avoid bias toward bigger or wider faces.
- Auto-Alignment: Reference sets began with 100 random objects per class and were reduced by removing low-quality objects before aligning the remaining objects.Each class used its reference set to align objects one by one.
- Automatic Assignment of Number of Orientation Classes: The number of orientation classes was automatically decided separately for each object category using configurations spanning 360, 180, and 90 degrees.The procedure evaluated these rotation-search configurations during reference-set generation.
Analysis
The analysis compares baseline and orientation-aware networks by tracking filter activations and dominant signal-flow paths across object rotations. ORION develops more orientation-specific responses and more distributed paths than the baseline.
- Filter Activations: Baseline weights were fine-tuned into ORION with a low learning rate to monitor corresponding filter changes.The baseline was first trained to a stable state without orientation outputs.
- Filter Activations: ORION filters became more sensitive to orientation-specific object features, including table legs in the exemplar analysis.Figure 6 shows activations for the same object across discretized rotations during the transition from baseline to ORION.
- Dominant Signal-Flow Paths: The dominant signal-flow path identifies highly contributing nodes layer by layer for a given object class and orientation.Paths are aggregated across objects by plotting histograms of selected node indices.
- Dominant Signal-Flow Paths: Baseline paths across class rotations often shared nodes, whereas ORION paths were more distributed across nodes.The difference was especially visible in convolutional layers and was interpreted as a result of orientation-boosting.
- Filter Activations: Figure 6 compares activation responses for one first-layer node across discretized input rotations and training-transition rows.Values below a threshold were removed to depict the 3D feature maps.
- Dominant Signal-Flow Paths: Figure 7(b) illustrates dominant paths for baseline and ORION across rotations of sample ModelNet10 object categories.The comparison focuses on how node usage differs between the two networks.
Extended Architecture
The extended architecture adds orientation-output nodes to the VoxNet-based network, with their number varying across experiments.
- The extended architecture is documented through details corresponding to Tables 1 and 2 of the main article.The supplied table caption identifies this as an architectural extension.
- The number of nodes dedicated to orientation output varies across experiments.The variation is explicitly marked with a dagger in the table.
- The table provides architecture details rather than a single fixed orientation-output size.Its caption specifically highlights experiment-dependent orientation-node counts.
11 Orientation Estimation Results
The paper reports orientation-class accuracies in Table 4, while noting that orientation estimation was only an auxiliary task. The extended architecture is the one introduced in the main article.
- The reported orientation estimator was trained as an auxiliary task rather than the primary objective.
- The authors note that orientation accuracy could improve further by emphasizing orientation estimation.
- Table 4 reports orientation estimation accuracies across different datasets.