PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Analysis of Deep Regression

Stéphane Lathuilière, Pablo Mesejo, Xavier Alameda-Pineda, Radu Horaud

arXiv:1803.08450v3cs.CV

TL;DR

Deep regression lacks systematic evaluation amid many architectures, preprocessing choices, and stochastic optimization effects. This paper statistically compares vanilla deep regression variants across vision problems and finds preprocessing often matters more than architecture, while tuned general-purpose networks can compete with specialized methods.

  • Problem

    Deep regression lacks systematic evaluation and statistical testing despite an overabundance of deep-learning studies and substantial implementation and preprocessing variability.

  • Method

    The paper systematically compares vanilla convolutional networks with linear regression layers and their variants across four vision problems using repeated runs, confidence intervals, and statistical tests.

  • Results

    Across the experiments, data preprocessing generally produces more variability than network modifications, and correctly fine-tuned deep regression networks compete with specialized methods in three of four problems.

  • Takeaways & Limitations

    General-purpose networks such as VGG-16 or ResNet-50 can achieve competitive results when adequately tuned, without requiring complex task-specific regression models.

  • Takeaways & Limitations

    State-of-the-art comparisons are uncertain because many published protocols lack detail and available results usually reflect average performance from possibly single runs.

Abstract

from arXiv · show

Deep learning revolutionized data science, and recently its popularity has grown exponentially, as did the amount of papers employing deep networks. Vision tasks, such as human pose estimation, did not escape from this trend. There is a large number of deep models, where small changes in the network architecture, or in the data pre-processing, together with the stochastic nature of the optimization procedures, produce notably different results, making extremely difficult to sift methods that significantly outperform others. This situation motivates the current study, in which we perform a systematic evaluation and statistical analysis of vanilla deep regression, i.e. convolutional neural networks with a linear regression top layer. This is the first comprehensive analysis of deep regression techniques. We perform experiments on four vision problems, and report confidence intervals for the median performance as well as the statistical significance of the results, if any. Surprisingly, the variability due to different data pre-processing procedures generally eclipses the variability due to modifications in the network architecture. Our results reinforce the hypothesis according to which, in general, a general-purpose network (e.g. VGG-16 or ResNet-50) adequately tuned can yield results close to the state-of-the-art without having to resort to more complex and ad-hoc regression models.

1 INTRODUCTION

Deep regression lacks systematic, statistically grounded comparisons despite widespread use and many competing architectures. This paper addresses the gap by evaluating vanilla deep regression across vision tasks, network variants, preprocessing strategies, and stochastic effects.

  • Background: Vanilla deep regression replaces a ConvNet’s classification softmax with a fully connected regression layer using linear or sigmoid activations.It is applied to continuous-value vision tasks including head-pose estimation, human pose estimation, and facial landmark detection.
  • Motivation: Researchers often lack statistical evaluations and rigorous methodological justification for deep-learning performance claims.Consequently, reported superiority can remain unclear and insufficiently statistically grounded.
  • Motivation: Prior work lacks a systematic evaluation of deep-learning advances in regression and contains an overabundance of rapidly introduced deep architectures.The paper argues that comparative empirical studies are needed to identify which components matter in deep regression.
  • Motivation: Single-run benchmarks and sparse preprocessing descriptions limit result reliability and reproducibility.Many published studies also omit implementation, practical, or data-preprocessing details.
  • Contribution: The study systematically compares vanilla deep regression using repeated experiments, confidence intervals, statistical tests, network variants, preprocessing strategies, and task-specific regression methods.Experiments cover head pose estimation, facial landmark detection, and full-body pose estimation.

2 EXPERIMENTAL PROTOCOL

The experimental protocol benchmarks vanilla deep regression with VGG-16 and ResNet-50 across three vision problems and four diverse datasets. Networks are adapted from ImageNet classification by replacing softmax with a target-sized linear regression layer.

  • Base architectures: The study uses VGG-16 and ResNet-50 as common base architectures for benchmarking network and preprocessing choices.VGG-16 is preferred over AlexNet, while ResNet-50 offers higher ImageNet performance and shorter training time.
  • Base architectures: Both pretrained networks replace the classification softmax with a linear regression layer whose output dimension matches the task target space.The models are initialized through ImageNet classification training before regression fine-tuning.
  • Datasets and tasks: Experiments address head-pose estimation, facial landmark detection, and human-body pose estimation using four datasets selected for diversity in output dimension, preprocessing, and augmentation requirements.The datasets are Biwi, combined LFW and NET, Parse, and MPII.
  • Datasets and tasks: Parse contains 305 images and uses training-only rotations from -12° to 12° in 0.5° steps for augmentation.The dataset annotates 14 body joints and challenges very deep architectures because of its small size.
  • Datasets and tasks: MPII is split into training, validation, and test subsets at 70%, 20%, and 10% because its official test set is unavailable for this study.Only mirroring is used for augmentation to keep the repeated statistical analysis tractable.
  • Computational protocol: The protocol summarizes more than 1000 experimental runs on a single Nvidia TITAN X configuration, with code and individual-run results made available online.The reported setup uses Keras 1.1.1 with the Theano 0.9.0 backend.

3 NETWORK OPTIMIZATION

The paper first evaluates optimizers and batch sizes to establish training choices before comparing network variants. Optimizer effects depend on the architecture and task, while batch-size sensitivity is stronger for VGG-16 than ResNet-50.

  • Optimization setup: The study treats network optimization and fine-tuning as synonymous because all models begin from ImageNet-pretrained weights.Optimization seeks the weights minimizing a loss that includes training performance and regularization terms.
  • Optimizer comparison: Adaptive optimizers modify learning rates using gradient histories: AdaGrad uses cumulative squared gradients, RMSProp uses recent exponentially weighted gradients, AdaDelta avoids manual learning-rate tuning, and Adam adds momentum.These mechanisms differ in how aggressively they adapt updates over time.
  • Optimizer comparison: The experiments compare AdaGrad, AdaDelta, Adam, and RMSProp using mini-batch gradient descent and mean square error loss.AdaDelta and Adam show the best overall training performance and convergence time.
  • Optimizer comparison: Optimizer choice can produce completely different converged loss values, and the best choice depends on the architecture and problem.With VGG-16, Adam performs best on FLD but is among the worst on MPII.
  • Batch-size comparison: Batch-size effects are more important for VGG-16 than ResNet-50.ResNet-50 yields comparable results across tested batch sizes, whereas VGG-16 benefits more from larger batches except on MPII.
  • Batch-size comparison: The subsequent experiments use batch sizes of 128 for VGG-16 and 64 for ResNet-50.A batch size of 128 could not be used with ResNet-50 because of GPU memory limitations on the Nvidia TITAN X.

4 STATISTICAL ANALYSIS OF THE RESULTS

The analysis accounts for stochastic effects in deep learning and uses non-parametric tests alongside 95% confidence intervals to compare configurations. It reports median-performance intervals while controlling error in multiple comparisons.

  • Statistical analysis: Stochastic effects arise from weight initialization, optimization, and regularization, complicating comparisons among architecture choices.These effects were observed in the optimization-technique and batch-size analyses.
  • Statistical analysis: The analysis uses non-parametric tests because performance measurements lack normality.Visual inspection and Lilliefors normality tests indicated non-normal data.
  • Statistical analysis: Wilcoxon signed-rank tests handle pairwise comparisons, while Bonferroni-Holm controls family-wise error for multiple comparisons.The Holm procedure is applied to multiple Wilcoxon tests when more than two methods are compared.
  • Confidence intervals: 95% confidence intervals are reported for the median mean absolute error to quantify the magnitude of each parameter of interest.The study emphasizes that p-values alone do not fully describe practical effect magnitude.
  • Confidence intervals: Confidence intervals for medians and other quantiles are estimated using the binomial distribution, with bounds given by ordered observations.The interval is formed between the jth and kth observations after rounding the calculated indices upward.

5 NETWORK VARIANTS

The study evaluates how optimization, normalization, dropout, fine-tuning depth, regression layers, and target/input representations affect vanilla deep regression across datasets and architectures. Results show that these choices are often dataset- and architecture-dependent, with heatmap regression consistently poor for vanilla regressors.

  • Experimental setup: The experiments compare network variants one choice at a time, using holdout validation and an independent test set to assess generalization.The study cannot evaluate all possible combinations because deep-network training is time-consuming.
  • Loss: Loss performance is dataset- and architecture-dependent, so the authors recommend trying MSE, MAE, and Huber loss on new datasets.The three losses are statistically equivalent for Parse, while significant differences occur on the other datasets.
  • Dropout: For VGG-16, dropout is generally recommended in both fully connected layers, although its effect is negligible on Parse and MPII.The 11−DO strategy is described as the safest option, while 00−DO should be avoided.
  • Target and input representations: Heatmap regression performs worse than alternative target/input representations for vanilla deep regression and is associated with poorer local minima.The paper notes that successful heatmap methods typically use more complex, ad-hoc, or cascaded structures.
  • Normalization: Normalization substantially benefits VGG-16, with layer normalization outperforming batch normalization in three of four experiments.For Biwi, batch normalization improves validation and test generalization despite not improving optimization.
  • Practical implications: When resources are limited, a suboptimal parameter choice may have little practical effect, but significantly worse configurations should be avoided.The paper distinguishes small performance differences from cases where train, validation, and test performance differ evidently.

6 DATA PRE-PROCESSING

The study compares image mirroring and dataset-specific preprocessing strategies for VGG-16 and ResNet-50, finding that preprocessing can substantially change performance and may interact with architecture.

  • Preprocessing protocols: Images are resized or geometrically transformed to common network inputs, with corresponding transformations applied to annotations.VGG-16 constrains inputs to 224 × 224 pixels; tested preprocessing includes cropping, resizing, padding, mirroring, and bounding-box expansion.
  • Dataset-specific strategies: For FLD, face bounding boxes are expanded by 0%, 5%, 15%, or 50%, whereas MPII uses only 0%, 5%, and 15% expansion because larger boxes may include multiple people.
  • Evaluation condition: Preprocessing strategies cannot be compared directly across transformations because both images and annotations are geometrically modified.The same pixel error can therefore have different meanings under different preprocessing strategies.
  • Results: For Biwi, 224 is best for VGG-16 and 64-Re for ResNet-50, while 224 is significantly worst for ResNet-50; their second-best confidence-interval differences are below 1 degree in MAE.
  • Results: Mirroring is consistently beneficial across datasets and networks, while preprocessing choices produce strong performance variations and require dataset-specific analysis.

7 POSITIONING OF VANILLA DEEP REGRESSION

The paper positions carefully tuned vanilla deep regression against task-specific state-of-the-art methods, finding competitiveness on some tasks but clear gaps on others, with cross-paper comparisons requiring caution.

  • Comparison goal: The comparison asks how closely correctly fine-tuned VGG-16 and ResNet-50 regression networks approach state-of-the-art methods without aiming to outperform task-specific systems.
  • Comparison limitation: State-of-the-art differences must be interpreted cautiously because external methods generally provide only average performance from incompletely specified protocols and likely single runs.
  • FLD: On FLD, the average run with ResNet-50’s optimal strategy is only 0.23% worse than the state of the art.Failure percentage counts errors larger than 5% of the bounding-box width.
  • Parse: Vanilla deep regression does not outperform the state of the art on Parse under strict PCP, which counts a limb correct below a 50% limb-length error threshold.
  • Biwi: Correctly fine-tuned deep regression competes well with state-of-the-art methods on Biwi, including a method using extra training data.
  • MPII: On MPII, state-of-the-art methods clearly outperform vanilla deep regression under PCKh, indicating benefits from more complex, task-specific approaches in that setting.

8 OVERALL DISCUSSION

Across configurations, network variants and preprocessing can produce substantial performance differences, with fine-tuning, regression loss, and preprocessing generally more influential than some architectural choices.

  • Configuration effects: Selecting network variants can improve performance by up to 2° MAE on Biwi, 14% error rate on FLD and MPII, and 6% success rate on Parse.
  • Architecture variability: VGG-16 generally shows larger improvement margins and more variable results than the generally more stable ResNet-50, with stated task-specific exceptions.
  • Critical factors: Fine-tuning, regression loss, and preprocessing are generally the most critical factors, whereas transfer-initialization and dropout show less variability between best and worst configurations.
  • Preprocessing effects: Preprocessing impact depends strongly on both the problem and architecture, so the optimal strategy can differ between networks on the same dataset.
  • Recommendations: The paper recommends comparing preprocessing strategies and reporting them transparently with statistical tests and, when possible, confidence intervals for median performance.

9 CONCLUSION

The paper presents a repeated-run statistical protocol for evaluating vanilla deep regression and concludes that careful preprocessing is central to competitive, interpretable results.

  • Contribution: Vanilla deep regression denotes convolutional neural networks with a linear regression layer and variants such as altered fine-tuning depth or preprocessing.
  • Evaluation protocol: Each configuration is run five times, with 95% confidence intervals for median performance and statistical tests used to distinguish stochastic effects from systematic improvements.
  • Main conclusion: Correctly fine-tuned deep regression competes with problem-specific methods in three of four problems.
  • Implications: The experiments emphasize transparent preprocessing descriptions and clean protocols to support meaningful, competitive, and reproducible conclusions.
Loading 1803.08450v3…