Source-linked AI summary
A Simple, Fast and Highly-Accurate Algorithm to Recover 3D Shape from 2D Landmarks on a Single Image
Ruiqi Zhao, Yan Wang, Aleix Martinez
TL;DR
Recovering 3D shape from landmarks in one image is difficult and ill-posed. The paper learns the 2D-to-3D mapping with a feed-forward deep network, reporting up to two-fold gains over prior state-of-the-art methods across rigid and deformable objects.
Problem
Recovering 3D geometry from 2D landmarks in a single image is difficult and ill-posed, despite humans doing it readily.
Source: §1 · p. 1Method
A feed-forward deep neural network learns a mapping from single-image 2D landmarks to 3D shape using hierarchical nonlinear transformations.
Source: §1 · p. 1Results
Across rigid, articulated, non-rigid, and highly-deformable objects, results are up to two-fold better than previous state-of-the-art methods.
Source: §5 · p. 7Takeaways & Limitations
The same network is used for cars, bodies, faces, and flags, can be trained with small sample sets, and runs faster than real time.
Source: §5 · p. 7The cited prior methods have boundaries: linear models lose accuracy for nonlinear mappings, and some algorithms do not work well across small and large training sets.
Source: §2 · p. 2
Abstract
from arXiv · showhide
Three-dimensional shape reconstruction of 2D landmark points on a single image is a hallmark of human vision, but is a task that has been proven difficult for computer vision algorithms. We define a feed-forward deep neural network algorithm that can reconstruct 3D shapes from 2D landmark points almost perfectly (i.e., with extremely small reconstruction errors), even when these 2D landmarks are from a single image. Our experimental results show an improvement of up to two-fold over state-of-the-art computer vision algorithms; 3D shape reconstruction of human faces is given at a reconstruction error < .004, cars at .0022, human bodies at .022, and highly-deformable flags at an error of .0004. Our algorithm was also a top performer at the 2016 3D Face Alignment in the Wild Challenge competition (done in conjunction with the European Conference on Computer Vision, ECCV) that required the reconstruction of 3D face shape from a single image. The derived algorithm can be trained in a couple hours and testing runs at more than 1, 000 frames/s on an i7 desktop. We also present an innovative data augmentation approach that allows us to train the system efficiently with small number of samples. And the system is robust to noise (e.g., imprecise landmark points) and missing data (e.g., occluded or undetected landmark points).
1 Introduction
The section frames single-image 3D landmark reconstruction as a difficult vision problem and proposes a deep network to learn the 2D-to-3D mapping.
1 Introduction
- The task is ill-posed, with challenges including generalizing across rigid and non-rigid objects, handling different training-set sizes, and running in real time.The introduction notes that existing approaches include single-image methods, while some alternatives require image sequences.Source: §1 · p. 1
- The proposed network uses hierarchical nonlinear transformations and is reported to outperform previous algorithms by a significant margin.It recovers depth from known 2D landmarks, and the 3D shape combines the network input and output.Source: §1 · p. 1
- The framework is intended for rigid or non-rigid objects with small or large training sets, including faces, cars, bodies, and flags.For small training sets, augmentation generates novel 2D landmark views across cameras, scales, and viewpoints.Source: §1 · pp. 1–2
- The algorithm runs at more than 1,000 frames/s on an i7 desktop, exceeding real-time speed.The stated speed is for testing.Source: §1 · p. 2, Figure 1 · p. 2
2 Related Work
Related work includes single-image shape-space methods and sequence-based reconstruction, with limitations in nonlinear shape modeling, category transfer, and training-set flexibility.
2 Related Work
- Single-image methods commonly fit shape-space models to infer 3D landmarks from 2D locations,,, [21].The section also describes an SfM-trained model that reconstructs shape from single-image landmarks.Source: §2 · p. 2
- Linear shape-basis methods lose accuracy when the true mapping is nonlinear and cannot efficiently handle highly deformable or articulated shapes.PCA and other linear methods are used to obtain low-dimensional shape bases,.Source: §2 · p. 2
- Some methods are category-specific: addresses human pose and focuses on car modeling, assumptions that do not typically generalize to other categories.Source: §2 · p. 2
- Existing algorithms can be poorly matched to training-set size: some do not improve with large datasets, while others fail to yield reasonable results with small datasets.Source: §2 · p. 2
- The authors state that their theoretical and experimental results resolve the limitations discussed in this section.Source: §2 · p. 2
3 Proposed Approach
The approach maps 2D landmarks from a single image to 3D, then addresses missing landmarks and limited training data through recurrent completion and camera-based data augmentation.
3.1 Preliminaries
- Under a weak-perspective camera model, the task is to recover 3D landmark coordinates from 2D landmarks, with the projected result defined up to scale.The approach standardizes variables to address this scale ambiguity.Source: §3.1 · p. 3
3.2.1 Proposed Neural Network.
- The multilayer network learns a standardized 2D-to-3D mapping by minimizing summed Euclidean distances between predicted and ground-truth landmark depths.Standardizing the coordinates removes the effects of 3D-object scaling and translation.Source: §3.2.1 · p. 3
3.2.2 Testing.
- At test time, standardized 2D landmarks are fed to the trained network to obtain depth, which is combined with the 2D coordinates to recover 3D shape up to scale.Source: §3.2.2 · p. 3
3.2.3 Missing Data.
- For missing landmarks, a recurrent layer estimates their 2D coordinates while the multilayer network predicts depth; a module merges both outputs into the final result.The recurrent layer uses weighted outputs across its iterations for intermediate supervision, and the identity function performed best in experiments.Source: §3.2.3 · pp. 3–4, Figure 2 · p. 4
3.2.4 Data augmentation approach.
- Camera-based augmentation generates views by varying camera parameters and adds Gaussian landmark noise with variance equivalent to about 3% of object size.The authors report very low reconstruction errors with small training sets; with large sets, augmentation can reduce error by incorporating camera parameters and detection errors underrepresented in the samples.Source: §3.2.4 · p. 4
3.2.5 Implementation Details.
- The feed-forward network has six layers, with 2n nodes in each of the first five and n in the last; training and validation sets are augmented, with early stopping after 10 iterations without validation improvement.The initial learning rate is .01.Source: §3.2.5 · p. 4
4 Experimental Results
Across the tested datasets, the algorithm is reported to recover 3D shape more accurately and efficiently than previous methods. The experiments cover human bodies, faces, cars, flags, and robustness to noisy or missing landmarks.
4 Experimental Results
- The method is significantly more accurate and efficient than previous approaches, with reconstruction error measured by scale-normalized Procrustes distance.Ground-truth coordinates in millimeters are normalized by the mean of all 3D landmark points.Source: §4 · p. 4
4.5 Noise and missing data
- The method is reported to tolerate moderate Gaussian landmark noise and to handle missing data.With one randomly missing landmark per sample, mean BU−3DFE reconstruction error is 0.006; CMU errors for subjects 13, 14, and 15 are 0.0413, 0.0396, and 0.0307, respectively.Source: §4 · p. 4, §4.5 · p. 7
4 Experimental Results
- Testing runs at more than 1,000 frames/s on a 3.40 GHz Intel Core i7 desktop.The experiments also include the CMU Motion Capture, BU−3DFE, FG3DCar, and flag databases.Source: §4 · p. 4
4.1 CMU Motion Capture Database
- CMU Motion Capture results show significantly greater accuracy than previously defined algorithms.The comparison is reported in Table 1.Source: §4.1 · p. 5, Table 1 · p. 5
4.2 3D Face Reconstruction
- The algorithm’s mean testing error was 0.004; using the same experimental setting, it outperformed the state-of-the-art method of in comparative results reported in Table 2.On a sequestered dataset to which the authors had no prior access, the algorithm was a top performer with a significant margin over other methods.Source: §4.2 · p. 6
4.3 FG3DCar Database
- The FG3DCar testing set has a mean 3D reconstruction error of 0.0022.Qualitative examples are shown in Figure 5; comparative results with are referenced, but no outcome is stated here.Source: §4.3 · p. 7
4.4 Flag Flapping in the Wind Database
- The flag sequence has a mean testing error of 0.0004, and reconstructed shapes are described as almost identical to ground truth.Figure 6 compares reconstructions with ground truth.Source: §4.4 · p. 7
5 Conclusions
The conclusion presents one feed-forward neural network for reconstructing 3D shapes from 2D landmarks across several object types. It reports very low errors, improvements of up to two-fold over prior methods, and testing above 1,000 frames/s.
5 Conclusions
- A single feed-forward neural network maps 2D landmarks to 3D shapes across rigid, articulated, non-rigid, and highly deformable objects.The conclusion reports results as much as two-fold better than previous state-of-the-art algorithms, testing above 1,000 frames/s, and training with small sample sets.Source: §5 · p. 7