Source-linked AI summary
Pretraining boosts out-of-domain robustness for pose estimation
Alexander Mathis, Thomas Biasi, Steffen Schneider, Mert Yüksekgönül, Byron Rogers, Matthias Bethge, Mackenzie W. Mathis
TL;DR
Pose estimation lacks direct robustness benchmarks for small datasets with many postures per individual and few individuals. The study benchmarks pose estimation on 30 Thoroughbred horses and 8114 frames with 22 labeled anatomical landmarks, including the corrupted Horse-C variant. Better ImageNet-performing architectures perform better within domain and on unseen horses when pretrained, with out-of-domain improvement of −0.93% per ImageNet-accuracy percentage point.
Problem
Pose estimation lacks direct robustness benchmarks for small datasets with many postures per individual and few individuals.
Method
The study benchmarks pose estimation on 30 Thoroughbred horses and 8114 frames with 22 labeled anatomical landmarks, including the corrupted Horse-C variant.
Results
Better ImageNet-performing architectures perform better within domain and on unseen horses when pretrained, with out-of-domain improvement of −0.93% per ImageNet-accuracy percentage point.
Takeaways & Limitations
ImageNet pretraining improves robustness to unseen horses and common corruptions, while stronger ImageNet architectures generalize better across the evaluated shifts.
Abstract
from arXiv · showhide
Neural networks are highly effective tools for pose estimation. However, as in other computer vision tasks, robustness to out-of-domain data remains a challenge, especially for small training sets that are common for real-world applications. Here, we probe the generalization ability with three architecture classes (MobileNetV2s, ResNets, and EfficientNets) for pose estimation. We developed a dataset of 30 horses that allowed for both "within-domain" and "out-of-domain" (unseen horse) benchmarking - this is a crucial test for robustness that current human pose estimation benchmarks do not directly address. We show that better ImageNet-performing architectures perform better on both within- and out-of-domain data if they are first pretrained on ImageNet. We additionally show that better ImageNet models generalize better across animal species. Furthermore, we introduce Horse-C, a new benchmark for common corruptions for pose estimation, and confirm that pretraining increases performance in this domain shift context as well. Overall, our results demonstrate that transfer learning is beneficial for out-of-domain robustness.
1. Introduction
The paper addresses whether pose-estimation models trained on few individuals generalize to unseen individuals. It introduces a 30-horse benchmark and shows that ImageNet pretraining improves out-of-domain robustness.
- Real-world pose estimation often uses only a few hundred labeled frames from roughly 10 individuals, unlike human benchmarks with many individuals but few postures per person.
- ImageNet pretraining gives a 2X boost on out-of-domain data compared with training from scratch.
- The benchmark contains 30 diverse Thoroughbreds with 22 labeled body parts across 8114 frames, enabling within-domain and held-out-horse evaluation.
- The dataset varies substantially in horse appearance, context, lighting, background, and relative size, creating a challenging robustness test.
- ImageNet performance predicts pose-estimation generalization within domain and on unseen horses, while sufficiently large task-specific training can match fine-tuning only within domain.
2. Related Work
Prior pose-estimation benchmarks provide limited evidence about generalization across individuals and robustness to realistic shifts. This work adds a multi-horse benchmark and evaluates transfer learning against related robustness and pretraining findings.
- Unlike common human benchmarks, prior animal and human datasets often provide few postures per individual, limiting evaluation of generalization across multiple postures and individuals.
- The new dataset contains 30 race horses with 22 body parts labeled in 8114 frames, supporting within-domain and out-of-domain generalization tests.
- The study confirms that task-specific training can match pretrained performance with sufficient data, but held-out individuals reveal a persistent advantage for transfer learning.
- The paper situates pretraining effects within evidence that the pretraining task matters, since large-scale pretraining can help classification while possibly harming localization.
- The work extends robustness evaluation beyond unseen horses by investigating common image corruptions, a setting previously not directly tested for pose estimation in the cited comparisons.
3. Data and Methods
The study builds horse and corruption benchmarks for evaluating pose-estimation generalization, then compares pretrained and randomly initialized convolutional backbones under matched training procedures.
- 3.1. Datasets and evaluation metrics: The Horse-30 dataset contains 8,114 frames from 30 horses with 22 annotated anatomical landmarks, split into Horse-10 training subsets and unseen-horse tests.Three splits use 10 randomly selected training horses, with 5% and 50% frame subsets evaluated on training, test, and out-of-domain horses.
- 3.1. Datasets and evaluation metrics: Horse-C applies 15 common corruptions at five severities to Horse-10, producing 75 dataset variants and more than 600,000 images.The corruptions include blurring, point-wise noise, and simulated weather conditions.
- 3.2. Architectures and Training Parameters: The experiments evaluate MobileNetV2, ResNet, and EfficientNet backbones within a consistent DeepLabCut pose-estimation pipeline.The toolbox was extended with MobileNetV2 and EfficientNet backbones while keeping data loading, training, and evaluation consistent.
- 3.2. Architectures and Training Parameters: Learning schedules and hyperparameters were cross-validated for pretrained and from-scratch models, with pretrained models trained for 30k iterations and scratch models for 180k.The schedules used cosine learning-rate decay, ADAM, batch size 8, and augmentation; cross-validation optimized out-of-domain performance.
- 3.4. Similarity Analysis: Representation similarity was assessed with linear centered kernel alignment across backbone depths for within-domain and out-of-domain images.The analyses compared ImageNet-pretrained and from-scratch variants and averaged results over three shuffles.
4. Results
Across horse identities, animal species, training regimes, and corruption shifts, stronger ImageNet architectures and ImageNet pretraining consistently improved out-of-domain pose-estimation robustness.
- 4.2. Generalization to novel horses: For unseen horses, ImageNet performance predicted pose-estimation performance more strongly out of domain: −0.93% error per ImageNet percentage point versus −0.33% within domain.Both regressions had R2 = 0.93; the out-of-domain result had p = 9 × 10−8 and the within-domain result p = 1.4 × 10−7.
- 4.3. Generalization across species: Better ImageNet architectures generally generalized better from other animal classes to held-out cows and sheep.The Animal Pose experiments trained on one or multiple species while holding out cow or sheep for evaluation.
- 4.4. Task-based training from scratch: From-scratch networks could match pretrained performance with enough data and time in some settings, but never caught up on novel-horse tests.With 5% training data, scratch models also failed to approach pretrained performance for larger architectures; stronger ResNets and EfficientNets generalized worse from scratch, unlike MobileNetV2s.
- 4.5. Common corruptions: Batch-normalization adaptation was crucial on common corruptions, while pretraining improved clean out-of-domain performance for MobileNetV2s and ResNets.The evaluation recomputed separate batch-normalization statistics for each horse and corruption type.
5. Discussion and conclusions
The benchmarks establish transfer learning as beneficial for pose-estimation robustness, while also introducing faster model variants and corruption testing. Pretraining improves generalization, although a substantial within-domain versus out-of-domain gap remains.
- Horse-10 and Horse-C provide benchmarks for out-of-domain horse identities and common image corruptions in pose estimation.Horse-C evaluates 15 corruptions across five severities.
- Adaptive batch normalization slightly improves out-of-domain performance for MobileNetV2 and ResNet models.
- Pretraining offers shorter training times and lower data requirements in addition to improved robustness and generalization.
- The introduced DeepLabCut variants retain high accuracy while providing up to double the inference speed of the original ResNet backbone.
- Pretrained ImageNet networks improve out-of-domain robustness by more than 2X over scratch-trained networks, but a 3-fold within-domain/out-of-domain performance gap remains.
A. Additional information on the Horse-10 dataset
The supplementary material documents dataset statistics, split assignments, model-search procedures, and Horse-30 comparisons. Learning schedules were cross-validated separately for pretrained and scratch-trained architectures, with different optimal choices for out-of-domain evaluation.
- The dataset tables report labeled frames, nose-to-eye scale, and within-domain or out-of-domain status for each shuffle.
- Cross-validation covered MobileNetV2, ResNet-50, and EfficientNet variants, while similar-network parameters were reused for other models.
- Training used ADAM, batch size 8, cosine learning-rate schedules, and image augmentation with random cropping and rotations.
- Pretrained models used cosine schedules without restarts as optimal for out-of-domain data, whereas scratch-trained models favored schedules with restarts.
- Horse-30 comparisons use three shuffles with 50% train/test splits and train models for 60,000 iterations using Horse-10 cosine schedules.
C. Performance (PCK per bodypart) for all networks on Horse-10
The supplementary performance material organizes PCK@0.3 results by body part for all evaluated architectures on within-domain and out-of-domain horses. It also defines abbreviations used for selected anatomical landmarks.
- The body-part abbreviations FF, HF, and HH denote front foot, hind foot, and hind hock, respectively.
- Tables 4 and 5 report PCK@0.3 percentages for several body parts across all evaluated architectures in within-domain and out-of-domain settings.
D. CKA analysis of training & trained vs. from scratch networks
CKA analyses compare representations across training conditions and network depths. The reported comparisons distinguish task-training effects from ImageNet initialization and evaluate representation similarity on within- and out-of-domain horses.
- Task training changes ResNet-50 representations relative to ImageNet-trained networks without horse pose-estimation task training.
- Figure 10 compares centered-kernel-alignment representations for scratch-trained and ImageNet-initialized networks on within-domain and out-of-domain horses.
- The CKA matrices average results over three data splits, with similarity ratios quantified as a function of network depth.
E. Results of within domain performance on Animal Pose
The study evaluates cross-species pose-estimation generalization by training on one animal species and testing on another, with within-domain baselines for cow and sheep.
- Cross-species evaluation trains on one species and tests on another, while within-domain baselines use 90% of each test species for training.
- The cow and sheep within-domain baselines are reported as normalized-error test-performance tables.
F. Full results on Horse-C
Horse-C evaluates pose estimation under 15 common image corruptions across five severities, using multiple architectures and training regimes. Batch normalization adaptation substantially improves corruption performance, while only slightly improving identity-shift performance and failing to eliminate the within-domain/out-of-domain gap.
- Horse-C contains 75 evaluation settings from 15 corruption types applied at five severities, totaling over 600,000 corrupted images.
- The benchmark evaluates MobileNetV2, ResNet-50, and EfficientNet variants trained either from scratch or by ImageNet pretraining and fine-tuning.
- Some corruptions, including elastic transform and pixelation, may alter the ground-truth posture itself.
- Batch normalization adaptation considerably improves prediction performance across all considered common corruptions.
- Batch normalization adaptation slightly improves performance across different horse identities but does not close the substantial within-domain/out-of-domain gap.
G. Inference Speed Benchmarking
The speed benchmark compares inference throughput for MobileNetV2, ResNet, and EfficientNet backbones across video resolutions, batch sizes, and dataset splits. MobileNetV2 is substantially faster than ResNet-50, especially for offline processing.
- Inference speed is evaluated on an 11,178-frame video at 512×512, 256×256, and 128×128 resolutions with batch sizes from 1 to 512.
- MobileNetV2 delivers more than 2× the inference speed of ResNet-50 for offline processing and about 40% higher speed at batch size 1 on a Titan RTX GPU.
- The benchmark covers MobileNetV2, ResNet, and EfficientNet architectures across three training-set shuffles.