Source-linked AI summary
Fixing the train-test resolution discrepancy
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, Hervé Jégou
TL;DR
The paper addresses the train-test distribution shift caused by different preprocessing and object scales under standard image-classification augmentation. It adjusts train and test resolutions and applies lightweight fine-tuning, achieving strong ImageNet accuracy, including 86.4% single-crop top-1 accuracy with a weakly supervised ResNeXt-101 32x48d.
Problem
Separately optimized training and testing preprocessing creates a significant distribution shift between the regimes, affecting classifier performance.
Method
The method adjusts crop resolutions and scales between training and testing and fine-tunes two layers to compensate for changed activation statistics.
Results
86.4% single-crop top-1 accuracy is obtained on ImageNet with a weakly supervised ResNeXt-101 32x48d pretrained on 940 million public images.
Takeaways & Limitations
The approach enables strong classifiers to train with smaller images while using higher test resolution and reducing training-time computation and memory.
Takeaways & Limitations
The imaging analysis assumes upright, fronto-parallel square objects and a camera field of view typically between 40° and 60°.
Abstract
from arXiv · showhide
Data-augmentation is key to the training of neural networks for image classification. This paper first shows that existing augmentations induce a significant discrepancy between the typical size of the objects seen by the classifier at train and test time. We experimentally validate that, for a target test resolution, using a lower train resolution offers better classification at test time. We then propose a simple yet effective and efficient strategy to optimize the classifier performance when the train and test resolutions differ. It involves only a computationally cheap fine-tuning of the network at the test resolution. This enables training strong classifiers using small training images. For instance, we obtain 77.1% top-1 accuracy on ImageNet with a ResNet-50 trained on 128x128 images, and 79.8% with one trained on 224x224 image. In addition, if we use extra training data we get 82.5% with the ResNet-50 train with 224x224 images. Conversely, when training a ResNeXt-101 32x48d pre-trained in weakly-supervised fashion on 940 million public images at resolution 224x224 and further optimizing for test resolution 320x320, we obtain a test top-1 accuracy of 86.4% (top-5: 98.0%) (single-crop). To the best of our knowledge this is the highest ImageNet single-crop, top-1 and top-5 accuracy to date.
1 Introduction
The paper identifies a distribution shift caused by separately optimized training and testing preprocessing, then proposes resolution adjustment and lightweight fine-tuning to compensate for it. This strategy preserves training augmentation while improving efficiency and ImageNet performance.
- 1 Introduction: Training and testing preprocessing create a significant distribution shift that harms test-time model performance.Training uses randomly sampled Regions of Classification, whereas testing uses central crops, despite identical input crop sizes.
- 1 Introduction: The method jointly optimizes train and test resolutions and scales while retaining the same Region of Classification sampling.It compensates for the shift in statistics caused by changing crop size.
- 1 Introduction: Fine-tuning only two layers provides a computationally cheap way to compensate for the changed crop-size statistics.This retains the benefits of existing training and testing preprocessing protocols, including training-data augmentation.
- 1 Introduction: Halving crop resolution reduces network evaluation cost threefold and significantly lowers memory consumption for a typical CNN.The efficiency benefit is especially relevant for GPU-based training.
- 1 Introduction: 86.4% single-crop top-1 accuracy is achieved on ImageNet with a weakly supervised ResNeXt-101 32x48d pretrained on 940 million public images.The model operates at much higher test resolution while retaining quick training.
2 Related work
Prior work establishes image classification as a central computer-vision benchmark and explores larger models, higher resolutions, augmentation, and pooling strategies. This paper’s setup is framed against differing train-test preprocessing and resolution practices.
- 2 Related work: Image classification serves as a benchmark for progress, and pretrained classifiers transfer to varied applications.Advances in image classification also translate to improved results on other tasks.
- 2 Related work: Recent ImageNet research improves performance using larger networks and higher-resolution images.The cited state-of-the-art ResNeXt-101 32x48d uses 224×224 training images, while EfficientNet-b7 uses 600×600 images.
- 2 Related work: Standard augmentation includes random-size crops, horizontal flips, and color jitter, while test-time augmentation can average predictions from multiple crops.Multiple test crops improve accuracy but require several forward passes per image.
- 2 Related work: Pooling methods such as p-pooling adapt networks to test resolutions substantially higher than the training resolution.These strategies are reported to improve performance and are also used in image retrieval.
- 2 Related work: Typical augmentation produces different Region-of-Classification area distributions during training and testing.The test-time distribution is spiky because center crops leave aspect-ratio variation as the main source of variability.
3 Region selection and scale statistics
CNN preprocessing changes both apparent object sizes and activation statistics, creating different train- and test-time input distributions. Larger test crops can improve accuracy despite these statistical shifts.
- RoC selection extracts a rectangular image region and resizes it to a square crop before CNN classification.
- Train-time scale augmentation: RandomResizedCrop varies training object scale by randomly sampling the RoC scale and resizing it to Ktrain × Ktrain.This augmentation is intended to expose the CNN to a wider range of object scales.
- Train-time scale augmentation: Preprocessing standardizes training apparent object size, making it independent of the original image resolution.Networks do not have built-in scale invariance, so this standardization affects the learned input distribution.
- Train-test mismatch: Training and testing preprocessing produce different apparent object sizes for the same image; under standard settings, test-time objects may appear as small as one-third their training size.The discrepancy arises from separately optimized train- and test-time protocols and sampled training scales.
- Activation statistics: Changing crop resolution strongly affects activation statistics after global average pooling, while convolutional receptive fields remain nearly unchanged.At 64 pixels, zero activations increase from 0.5% to 29.8%, and values above 2 increase from 1.2% to 11.9%.
- Larger test crops: 78.4% accuracy at Ktest = 288 exceeds 77.0% at the native Ktest = Ktrain = 224, with higher test resolutions generally performing better.Increasing test crop size reduces the train-test object-size mismatch but also changes activation statistics.
4 Method
The method corrects the train-test discrepancy in object scale and activation statistics by increasing test crop size and adapting the network before global pooling. It uses lightweight adaptation while retaining existing preprocessing benefits.
- The method removes apparent object-size differences by increasing the crop size at test time.
- Keeping Ktest/Ktrain constant while increasing test crop size requires Ktest > Ktrain, which skews activation statistics.The method therefore addresses both object scale and activation-statistics changes.
- Parametric adaptation: Parametric adaptation maps the new average-pooling output distribution back to the original distribution with a scalar activation transformation.It provides measurable but limited accuracy improvement because the model does not distinguish distributions across pooled components.
- Adaptation via fine-tuning: Fine-tuning adapts the model on the same training set after switching from Ktrain to Ktest, restricting updates to the last layers.
- Adaptation via fine-tuning: Fine-tuning includes the batch normalization before global pooling and uses test-time augmentation to adapt sparsity and avoid further domain shifts.
- Adaptation via fine-tuning: After fine-tuning, pooling activation statistics closely resemble train-time statistics, although this similarity does not by itself imply higher accuracy.
5 Experiments
Experiments show that increasing test resolution improves ResNet-50 accuracy, while resolution adaptation further improves performance and permits efficient low-resolution training. The approach also scales to larger networks and offers favorable training-time trade-offs.
- Resolution effects: 77.0% to 78.4% top-1 accuracy results when a ResNet-50 trained at resolution 224 is tested at higher resolution without adaptation.Increasing test resolution raises accuracy across the evaluated networks.
- Resolution adaptation: 79% top-1 accuracy is achieved by fine-tuning the classifier and last batch-normalization layer of a ResNet-50 trained at Ktrain = 224.The maximum accuracy shifts from Ktest = 288 without fine-tuning to Ktest = 384 after fine-tuning.
- Multiple resolutions: 79.5% single-crop accuracy is improved from 79.0% using Ktrain = 224 and Ktest = [384, 352].The method averages classification scores across several test resolutions, increasing inference time modestly while keeping training time unchanged.
- Larger networks: 86.4% top-1 accuracy is reached by ResNeXt-101 32x48d when increasing the test resolution to Ktest = 320.For PNASNet-5-Large, Ktest = 480 increases accuracy by 1 percentage point, and a 10-crop ensemble reaches 83.9%.
- Speed-accuracy trade-off: 2.3× faster training is reported for a fine-tuned network at Ktest = 384 than for a network trained natively at that resolution.In the Ktrain = 128 regime, fine-tuning raises training time from 111.8 h to 124.1 h (+11%), while achieving 77.1% accuracy versus 133.9 h for native-resolution training.
6 Conclusion
The paper finds that adjusting crop resolution and applying lightweight parameter adaptation can improve classifier accuracy while preserving efficient training. It also reports applications to larger networks and transfer-learning evaluations.
- Adjusting crop resolution and lightweight parameter adaptation significantly improve standard classifier accuracy, with otherwise unchanged conditions.
- The approach is presented as a way to correct networks trained at 224 × 224 resolution after training.
Supplementary material for “Fixing the train-test resolution discrepancy”
The supplementary material provides additional analyses, experiments, and results that were not included in the main paper.
- The supplement covers activation-distribution estimation, image-size border and round-off effects, exhaustive result tables, and fine-grained recognition challenges.
A.1 Parametric Fr´echet model after average-pooling
This section models post-pooling activation statistics with a Fréchet distribution and tests an affine transformation to correct resolution-induced discrepancies. The scalar distribution-based correction does not improve accuracy over the baseline.
- ReLU outputs combine a cropped unit Gaussian with a point mass at zero, while average pooling sums these distributions.
- The empirical post-pooling distribution is close to an extreme-value distribution because only the positive Gaussian tails contribute to outputs.
- The post-pooling activation distribution is modeled as a two-parameter Fréchet, or inverse Weibull, distribution.
- The correction estimates reference and target-resolution distribution parameters, then applies an affine scaling while ignoring zero activations.
- The scalar distribution model fails to improve baseline accuracy because it does not capture individual activation distributions or convolutional-trunk behavior.
A.2 Gaussian model before the last ReLU activation
This section studies channel-wise activation distributions before the final ReLU and adapts them across resolutions. Batch-normalization fine-tuning performs better overall than direct distribution adjustment, while resolution changes also create discrete border effects.
- Higher resolutions shift pre-ReLU channel distributions toward lower means and smaller variances.
- Matching pre-ReLU distributions can alter post-pooling sparsity, unlike the earlier Fréchet-based analysis.
- Batch-normalization fine-tuning generally outperforms distribution-based adaptation, especially beyond small resolutions.
- Accuracy is not monotonic in input resolution because convolutional kernel sizes and strides create discrete border effects.
- Accuracy drops occur at multiples of 32, so the experiments use resolutions that are multiples of 32.
C Result tables
The supplementary material provides full result tables covering resolution combinations, ablations, runtime, test-time augmentation comparisons, and a fixed-size random-crop variant.
- Table 5 reports top-1 validation accuracy across training and testing resolutions, with and without the data-driven adaptation strategy and test-time augmentations.
- Table 6 contains an ablation study of training-time and test-time data augmentation components.
- Table 7 reports training and fine-tuning execution times, including the total time for both stages.
- Tables 8 and 9 compare test-time augmentation variants and resolution combinations using fixed-size random cropping instead of random resized cropping.
D Impact of Random Resized Crop
The section evaluates replacing RandomResizedCrop with resize followed by fixed-size random cropping and also describes competition experiments using the proposed approach.
- Impact of Random Resized Crop: Replacing RandomResizedCrop with resize followed by fixed-size random cropping tests whether the previously observed effect depends mainly on RandomResizedCrop.
- Competition datasets: The iNaturalist Challenge 2019 dataset contains 1,010 animal and plant species, with 268,243 training images and 35,351 test images.
- Competition method: The competition systems averaged predictions from four different CNN architectures, each using 10 image crops.
- Competition method: The competition experiments used two fine-tuning stages: dataset adaptation followed by higher-resolution adaptation.
E.3 Results
The reported competition submissions used the proposed method off-the-shelf and achieved results within three points of the winners while using fewer evaluations and smaller ensembles.
- The top-performing approach finished 3 and 1 points behind the winners in the iNaturalist and Herbarium competitions, respectively.
- The submissions used five public-test evaluations for iNaturalist and eight for Herbarium, fewer than the best-performing competing approaches.
- The ensembles combined fewer CNNs than the two best-performing competing ensembles.