Source-linked AI summary
How many images do I need? Understanding how sample size per class affects deep learning model performance metrics for balanced designs in autonomous wildlife monitoring
Saleh Shahinfar, Paul Meek, Greg Falzon
TL;DR
The paper addresses limited evidence about how many images per species are needed to train accurate wildlife camera-trap classifiers. It evaluates progressively larger balanced training sets across deep-learning architectures and datasets, finding logarithmic performance relationships and providing practitioner-oriented predictive models.
Problem
Limited camera-trapping evidence establishes how increasing images per species affects deep-learning accuracy, making advance training-sample planning difficult.
Method
The study compares balanced training sets of 10–1000 images per class across six deep-learning architectures, three datasets, and two tuning schemes, using regression models and GAMs.
Results
Accuracy, precision, and true positive rate increased logarithmically while false positive rate decreased inversely logarithmically; improvements slowed after 150 images and became insubstantial after 500.
Takeaways & Limitations
For balanced long-term monitoring studies, 150–500 images per class may provide reasonable classification accuracy, with simple regressions offering an easy a priori guide.
Takeaways & Limitations
The conclusions are most relevant to long-term monitoring sites with visually similar backgrounds and may not extend to surveys across diverse random sites, where algorithm robustness can fail.
Abstract
from arXiv · showhide
Deep learning (DL) algorithms are the state of the art in automated classification of wildlife camera trap images. The challenge is that the ecologist cannot know in advance how many images per species they need to collect for model training in order to achieve their desired classification accuracy. In fact there is limited empirical evidence in the context of camera trapping to demonstrate that increasing sample size will lead to improved accuracy. In this study we explore in depth the issues of deep learning model performance for progressively increasing per class (species) sample sizes. We also provide ecologists with an approximation formula to estimate how many images per animal species they need for certain accuracy level a priori. This will help ecologists for optimal allocation of resources, work and efficient study design. In order to investigate the effect of number of training images; seven training sets with 10, 20, 50, 150, 500, 1000 images per class were designed. Six deep learning architectures namely ResNet-18, ResNet-50, ResNet-152, DnsNet-121, DnsNet-161, and DnsNet-201 were trained and tested on a common exclusive testing set of 250 images per class. The whole experiment was repeated on three similar datasets from Australia, Africa and North America and the results were compared. Simple regression equations for use by practitioners to approximate model performance metrics are provided. Generalized additive models (GAM) are shown to be effective in modelling DL performance metrics based on the number of training images per class, tuning scheme and dataset. Key-words: Camera Traps, Deep Learning, Ecological Informatics, Generalised Additive Models, Learning Curves, Predictive Modelling, Wildlife.
1. Introduction
Camera trapping produces large image volumes that make manual analysis difficult, motivating automated deep-learning classification. A central unresolved practical question is how many images per species are needed for target performance.
- Camera trapping has created a gap between large-scale image capture and manual data storage and analysis.
- Deep-learning algorithms, particularly convolutional neural networks, are a promising route for automated animal detection and species classification.
- Class imbalance can bias CNN models because common species contribute many more images than rare or endangered species.
- The study addresses limited camera-trapping evidence about how training images per species relate to expected model performance metrics.
2.1. Data
The study compares balanced, subsampled camera-trap datasets from Australia, Africa, and North America. Sampling addresses strong spatio-temporal correlation that can make images visually repetitive and reduce realistic variation.
- Three datasets from Australia, Africa, and North America were used to provide geographically and taxonomically diverse locations.
- The Australian dataset contains 29 native-animal and livestock species collected along forestry trails in New South Wales.
- Snapshot Serengeti contains 7 million images, including 1.2 million with animals, spanning 54 African wildlife species.
- Snapshot Wisconsin covers all Wisconsin counties and includes 43 distinct animal species.
- Each dataset was randomly subsampled into eight classes: six animal species, an empty-image class, and an others category.
- High spatio-temporal correlation creates visually similar sequences, so subsampling was used to improve variation in model data.
2.2. Deep Learning Models
The study evaluates ResNet and DenseNet architectures using transfer learning, alternative tuning and augmentation schemes, and four per-class performance metrics. Training uses labelled images and stochastic optimization, while model selection is based on validation loss.
- The experiment uses three ResNet variants and three DenseNet variants to compare deep-learning architectures.
- Transfer learning adapts a model trained on a related task, reducing the labelled data and computational effort required for a target CNN.
- Training Scheme: Shallow tuning modifies only final fully connected layers, whereas deep tuning permits modification of all or several pretrained layers.
- Training Scheme: Models are trained with labelled data and stochastic optimization, using ADAM to optimize CrossEntropy loss.
- Training Scheme: Four augmentation schemes vary whether training and testing images receive augmentation, generating pseudo-repetitions for later regression modelling.
- Model Performance Metrics: The evaluated metrics are per-class accuracy, precision, true positive rate, and false positive rate.
2.3 Predictive Modelling
The study uses ordinary least squares models for simple a priori performance estimates and generalized additive models to model nonlinear effects of training-image count and other predictors. Separate GAMs model each performance metric, with dataset-specific smooth effects for training-image number.
- Ordinary least squares models estimate expected model performance metrics for a specified training sample size.
- Generalized additive models extend generalized linear models by representing the linear predictor as a sum of smooth covariate functions.
- Predictors include tuning scheme, dataset, six network architectures, and six training-image levels: 10, 20, 50, 150, 500, and 1000 images.
- Backward elimination removed nonsignificant main effects and interactions at the 5% significance level, while the final model nests a cubic-spline training-image effect within dataset.
- The analysis models ACC, PRC, TPR, and FPR separately, using a Beta regression family with a logit link because performance-metric distributions resembled Beta distributions.
- A separate FPR model omits tuning because backward elimination found tuning nonsignificant for that metric.
3. Results
The results present model-performance metrics across datasets, architectures, training-set sizes, augmentation scenarios, and tuning schedules, alongside regression-based predictive models.
- 3. Results: Performance metrics were aggregated across 864 models spanning three datasets, six training-set sizes, six network architectures, four augmentation scenarios, and two tuning schedules.The reported metrics were accuracy (ACC), precision (PRC), true positive rate (TPR), and false positive rate (FPR).
- 3. Results: Ordinary Least Squares models provide a priori estimates of model performance metrics from training-sample size, while GAMs offer more detailed predictive modelling.The regression models are summarized in Tables 1–3 for the reported performance metrics.
- 3. Results: Figure 3 compares realised performance metrics with Ordinary Least Squares predictions as the number of training images changes.Blue dots represent realised metrics, while red lines and dots represent predicted metrics.
4. Discussion
Across balanced datasets, increasing training images generally improved deep-learning metrics logarithmically, with gains slowing after 150–500 images per class. Dataset, architecture, and tuning choices also shaped performance and variability.
- Limitations: The findings should not be routinely generalized to other camera-trap scenarios because available approaches may produce unpredictable and unreliable results.The study’s current investigation concerns larger-scale, longer-term projects that can invest in adequate samples and model development.
- Effect of increasing training images: ACC, PRC, and TPR improved logarithmically with training-set size, while FPR decreased inversely logarithmically across balanced datasets.From 10 to 1000 images per class, ACC increased 4–14%, PRC 15–63%, TPR 12–66%, and FPR decreased 50–83%.
- Effect of increasing training images: 150–500 images per class were sufficient for reasonable project-specific classification accuracy because performance gains slowed after 150 images and became insubstantial after 500.
- Effect of dataset: Australian and Serengeti datasets changed rank as sample size increased, whereas Wisconsin performed poorest, likely because many images were nocturnal and animals were difficult to identify.The authors note that the Australian–Serengeti difference could reflect either a real effect or random sampling.
- Effect of network architecture: DenseNet slightly outperformed ResNet, DenseNet-161 had the highest performance, and ResNet-18 the lowest; metric variance decreased two- to fourfold as images increased.The variance reduction was negligible for deeper networks.
- Effect of transfer learning scheme: Shallow tuning was adequate and competitive with deep tuning, although the difference was not statistically significant.The authors suggest insufficient epochs and transfer from ImageNet as possible explanations for deep tuning’s inferior performance.
- Generalized additive models: GAM analyses found significant associations of performance metrics with tuning scheme, dataset, architectures, and training-image count nested within dataset.FPR was the exception for tuning-scheme association.
- Practical guide for practitioners: Simple logarithmic regression models were less well fitted than GAMs but offered an easy guide for estimating performance before data collection.The practical guidance also recommends distinctive, non-burst-sequence images, shallow fine-tuning, and deeper networks where possible.
5. Conclusions and Future Works
The study addresses how much labelled data is needed for reliable wildlife-image recognition, evaluating training-image counts, training schemes, and architectures across geographically diverse datasets. Its logarithmic sample-size relationship is most relevant to long-term monitoring sites with visually similar backgrounds, while broader settings require further study.
- Camera-trap automation could reduce staff workload and support monitoring at previously unattainable scales, but reliable quality control requires sufficient labelled images per species.
- The study evaluates training-image counts, training schemes, and network architectures across three geographically and taxonomically diverse camera-trap datasets.
- The logarithmic relationship between training images and performance is most relevant to long-term monitoring sites with highly visually similar image backgrounds.
- Recognition robustness in random-site surveys and the effects of incorrect labelling, imbalanced samples, image quality, camera differences, and class composition require further research.
Appendix A:
Appendix A presents model-performance trends and test metrics across training-image counts, network architectures, and three wildlife datasets.
- Figures: Figure A1 shows overall accuracy trends as the number of training images varies across deep learning algorithms and Australia, Serengeti, and Wisconsin datasets.Each plotted point represents an observation in the trend graph.
- Figures: Figure A2 presents precision trends by training-image count, deep learning algorithm, and Australia, Serengeti, and Wisconsin dataset.The graph uses plotted points to represent observations.
- Figures: Figure A3 reports recall, identified as true positive rate, across training-image counts, algorithms, and the three datasets.The appendix labels recall as true positive rate.
- Figures: Figure A4 reports false positive rate trends across training-image counts, algorithms, and the three datasets.The figure organizes the trend by algorithm and dataset.
- Tables: Tables A2–A4 report accuracy, precision, true positive rate, and false positive rate on test sets for Australia, Serengeti, and Wisconsin.Metrics are stratified by class and training-image count, then aggregated across architecture, tuning, and augmentation strategy; thresholds are marked in bold.
- Tables: Table A5 reports accuracy, precision, true positive rate, and false positive rate across network architectures and training-image counts.These metrics are aggregated across datasets, classes, tuning, and augmentation strategy, with specified thresholds highlighted.