Source-linked AI summary
Recombinator Networks: Learning Coarse-to-Fine Feature Aggregation
Sina Honari, Jason Yosinski, Pascal Vincent, Christopher Pal
TL;DR
Precise localization needs spatial detail, while pooling provides robustness by discarding that detail. Recombinator Networks condition fine features on coarse features early and add denoising, improving facial keypoint results over summation-based models and prior state of the art without extra data.
Problem
Pooling improves robustness but discards spatial precision, creating a challenge for pixel-level localization and segmentation.
Method
Recombinator Networks feed upsampled coarse features into finer branches during feature formation, trained end-to-end, and combine them with a learned denoising model.
Results
The model improves on summation-based architectures, reduces prior state-of-the-art error by 15% on the 300W IBUG subset, and reports state-of-the-art performance on facial keypoint benchmarks.
Takeaways & Limitations
Early coarse-to-fine feature interaction improves facial keypoint prediction, while denoising helps handle more complex cases such as occlusions.
Takeaways & Limitations
The test sets differ substantially from training data, with more occlusion, extreme rotation, pose, and expression cases.
Abstract
from arXiv · showhide
Deep neural networks with alternating convolutional, max-pooling and decimation layers are widely used in state of the art architectures for computer vision. Max-pooling purposefully discards precise spatial information in order to create features that are more robust, and typically organized as lower resolution spatial feature maps. On some tasks, such as whole-image classification, max-pooling derived features are well suited; however, for tasks requiring precise localization, such as pixel level prediction and segmentation, max-pooling destroys exactly the information required to perform well. Precise localization may be preserved by shallow convnets without pooling but at the expense of robustness. Can we have our max-pooled multi-layered cake and eat it too? Several papers have proposed summation and concatenation based methods for combining upsampled coarse, abstract features with finer features to produce robust pixel level predictions. Here we introduce another model --- dubbed Recombinator Networks --- where coarse features inform finer features early in their formation such that finer features can make use of several layers of computation in deciding how to use coarse features. The model is trained once, end-to-end and performs better than summation-based architectures, reducing the error from the previous state of the art on two facial keypoint datasets, AFW and AFLW, by 30\% and beating the current state-of-the-art on 300W without using extra data. We improve performance even further by adding a denoising prediction model based on a novel convnet formulation.
1. Introduction
Pooling creates robust, increasingly global features but discards spatial precision needed for pixel-level localization. Recombinator Networks address this by conditioning fine features on coarse features early, and add denoising to improve keypoint predictions.
- Max-pooling and decimation improve robustness and translation invariance but discard precise spatial information needed for pixel-level localization.
- Existing coarse-to-fine methods aggregate detection paths only at the end, limiting how finely localized features can use coarse scene information.
- Recombinator Networks condition finely localized features on higher-level coarse features during their formation.
- The paper introduces a coarse-to-fine architecture, a denoising prediction model, an evaluation of architectural variants, and state-of-the-art facial keypoint results.
2. Related work
Related work combines representations from multiple resolutions for localization and prediction tasks. This paper adapts that strategy into a SumNet baseline while distinguishing RCN through earlier feature interaction and learned denoising.
- FCNs, Hypercolumns, and related models upsample coarse features and add or concatenate them with fine features for multiresolution prediction.
- These multiresolution approaches have improved results in detection, segmentation, and human-body pose estimation.
- The paper’s SumNet baseline concatenates features across hierarchy levels and applies a weighted sum before final softmax predictions.
- U-Net merges features across levels, while the paper notes that these related models do not use its learned denoising post-processing.
3. Summation versus Recombinator Networks
SumNet merges independently processed multiresolution branches only before prediction, whereas RCN passes upsampled coarse features into finer branches for further computation. RCN is trained end-to-end, with skip connections extending information flow across all coarser branches.
- Summation based Networks: SumNet preserves resolution within horizontal branches, upsamples each branch to 80 × 80, and combines their outputs with a weighted sum before softmax.
- Summation based Networks: SumNet branches communicate through output-layer updates, so their features are merged linearly only at the end of processing.
- The Recombinator Networks: RCN upsamples each branch and concatenates it with the next finer branch, allowing finer layers to process coarse features rather than merely sum final maps.
- The Recombinator Networks: The model is trained globally by backpropagation, while an L2 regression objective performed worse than the softmax probability-map formulation.
- The Recombinator Networks: RCN with skip connections concatenates each branch with upsampled outputs from all coarser branches and converges faster while outperforming SumNet.
4. Denoising keypoint model
The denoising model addresses occluded keypoints and implausible joint configurations that can arise from convolutional predictions. Its output is combined with RCN scores to produce final probability maps.
- Convolutional predictors may select nearby edges for occluded keypoints and produce probability maps inconsistent with plausible keypoint distributions.
- The denoising model is trained using corrupted keypoints and a loss applied to those corrupted predictions.
- At inference, denoising and RCN pre-softmax scores are summed before softmax to generate final keypoint probability maps.
5. Experimental setup and results
The evaluation tests SumNet and Recombinator Networks on facial keypoint datasets under differing train–test conditions, branch configurations, and preprocessing choices. RCN benefits from multi-resolution feature merging and achieves strong results against prior models, including a 15% lower IBUG error than CFSS without extra data.
- 5. Experimental setup and results: The experiments evaluate normalized Euclidean landmark error on AFW, AFLW, and 300W, using MTFL-derived training and established test subsets.AFW and AFLW use five keypoints, while 300W contains 68 keypoints with common and challenging subsets.
- 5. Experimental setup and results: The test sets are substantially harder and distribution-shifted relative to training, especially through occlusion, extreme rotation, pose, and expression.Training augmentation includes jittering and, for the five-keypoint experiments, random rectangular occlusions to bring training conditions closer to testing conditions.
- 5.1. Evaluation on SumNet and RCN: The best branch ablation uses all four coarse, medium, and fine resolutions, while coarsest-plus-finest branches alone provide reasonable performance.The finest branch alone performs poorly because of many false positives, whereas the coarsest branch alone is reasonable but less accurate.
- 5.1. Evaluation on SumNet and RCN: The best expanded model uses six branches, and RCN converges faster than SumNet during training.The six-branch result is reported alongside comparisons involving branch count, occlusion preprocessing, and skip connections.
- 5.2. Comparison with other models: RCN improves the SumNet baseline on AFW and AFLW and converges faster than both the baseline and compared models.SumNet is described as equivalent to FCN and Hypercolumn models in this comparison.
- 5.2. Comparison with other models: 15% lower error on the IBUG subset than CFSS was achieved without using extra data.The 300W comparison uses the current state-of-the-art model without extra data as the reference.
6. Conclusion
The paper introduces Recombinator Networks for combining pooled coarse features with fine non-pooled features, improving on summation-based approaches in training time and facial keypoint accuracy.
- Recombinator Networks combine coarse pooled feature maps with fine non-pooled features in convolutional neural networks.The architecture feeds coarser branches into finer branches during feature formation.
- Feeding coarser branches into finer branches lets finer resolutions learn from features extracted by coarser branches.
- The architecture improves on previous summation-based approaches through reduced training time and increased facial keypoint prediction accuracy.
Supplementary Information for Recombinator Networks: Learning Coarse-to-Fine Feature Aggregation
The supplementary figures visualize branch activations in SumNet and RCN and compare their performance across increasingly difficult merged AFW and AFLW test examples.
- Figure S1 compares pre-sum and softmax activations across the coarsest-to-finest branches of a four-branch SumNet model.It also shows the summed SumNet branches, RCN pre-softmax values, and true keypoint locations; the displayed softmax values are illustrative rather than trained-model components.
- Figure S1 shows the summed branches in SumNet alongside pre-softmax values in RCN, with true keypoint locations marked for correspondence.
- Figure S2 evaluates SumNet and RCN with and without occlusion preprocessing on merged AFW and AFLW test sets as example difficulty increases.Lower error is better; image difficulty is estimated from the average error across the four models.