Source-linked AI summary
Camera trap classification with deep learning under ground truth uncertainty
Leonard Hockerts, Peter S. Stewart, Sarthak Arora, Tiffany J. Vlaar
TL;DR
Supervised ecological image classification relies on costly annotation, often replaced by volunteer labels whose disagreement creates uncertain ground truth. Using two camera-trap datasets with volunteer and expert classifications, the study tests the effects of training with that uncertainty and finds improved test accuracy, especially on difficult volunteer-classified images, while cross-dataset generalisation did not improve.
Problem
Costly annotation motivates volunteer labels, but the effects of their disagreement on model accuracy across difficulty, species, datasets, and class imbalance are not well understood.
Method
The study uses two camera-trap datasets with volunteer and expert classifications to compare deep-learning training under different levels of ground truth uncertainty.
Results
Training under higher ground truth uncertainty improved overall test classification accuracy, particularly for images difficult for volunteers, but did not improve generalisation to a different dataset.
Takeaways & Limitations
Including moderate label disagreement and using ImageNet-pre-trained models can improve accuracy, especially on difficult ecological images.
Takeaways & Limitations
Accuracy gains on unseen images from the same dataset did not extend to generalisation to a different dataset.
Abstract
from arXiv · showhide
Supervised deep learning methods enable the rapid processing of ecological image data, but depend on a costly annotation process. Consequently, training labels are commonly derived from volunteer citizen science projects. However, disagreement among volunteers introduces uncertainty in the "ground truth" data that are assumed to be correct for model training and validation. Using two datasets containing camera trap images with associated volunteer and expert classifications, we investigated the effects of training under higher ground truth uncertainty. We observed improved overall test accuracy, particularly for images that were more difficult for volunteers. Species-level accuracy also generally improved, but generalisation to a different dataset did not. The benefits of ground truth uncertainty were enhanced by pre-training on ImageNet. Pre-training also reduced the number of training epochs required; further reductions in computational cost, but not gains in accuracy, resulted from additional pre-training on other camera trap images. With unbalanced training data, we still observed a clear benefit of increased ground truth uncertainty for overall accuracy, especially on difficult images. Class imbalance improved accuracy for common species, reduced rare species accuracy, and changed patterns of misclassification to more closely resemble mistakes made by volunteers. Our findings have implications for applying deep learning across ecological image types with multiple labels. Practitioners can improve accuracy, especially on difficult examples, by including moderate levels of label disagreement during training and using models pre-trained on general image data. In addition to improving the use of citizen science-derived labels in model training, our study suggests avenues for more effectively integrating human and deep learning classifications in combined workflows. (abridged)
1 Introduction
Ecological deep-learning classification can process large image datasets, but supervised models depend on costly labels often obtained through volunteer projects. This study examines how realistic volunteer disagreement affects accuracy, difficult images, generalisation, and class imbalance.
- Supervised ecological image classification requires costly annotation, motivating the use of volunteer citizen-science labels.
- Lowering volunteer-agreement thresholds retains images without unanimous agreement but introduces uncertainty into labels used to train and validate models.
- Disagreement may expose models to more difficult examples and increase available training data, while also risking incorrect labels that damage performance.
- The overall effect of ground truth uncertainty across image difficulty, species, datasets, and class balance remains poorly understood.
- The study uses two camera-trap datasets with volunteer and expert labels to address four questions about difficult images, species accuracy, cross-dataset generalisation, and class imbalance.
2.1 Datasets
The study uses two open-access camera-trap datasets with volunteer and expert classifications: Prickly Pear Project Kenya and Snapshot Serengeti. They differ in size, setting, species composition, and class balance.
- The analysis combines Prickly Pear Project Kenya and Snapshot Serengeti, both containing camera-trap images with volunteer and expert classifications.
- Prickly Pear Project Kenya: The Kenya images were classified by 8290 Zooniverse volunteers with varied experience.
- Snapshot Serengeti: Snapshot Serengeti contains more than 40 mammal and bird species photographed in Tanzania’s Serengeti National Park.
2.2 Image and label preparation
Images were filtered and standardised before training, while labels were generated from volunteer agreement thresholds and expert classifications. The resulting datasets retained multiple levels of ground truth uncertainty.
- The study focused on the 17 most common Kenya species and 16 most common Snapshot Serengeti species.
- Image filtering: Images with insufficient classifications, non-focal consensus species, or multiple recorded species were excluded.
- Image filtering: Snapshot Serengeti sequences were reduced to the middle image of each capture event, whose sequences contained 1–12 images.
- Image processing: Images were cropped to remove metadata and logos, then resized to dataset-specific dimensions.
- Label preparation: Training labels used majority-vote acceptance thresholds of 100%, 90%, or 66% agreement, while expert classifications supplied test labels.
2.3 Model architecture and implementation
Experiments used ResNet-50 models implemented in PyTorch, with augmentation, Adam optimisation, learning-rate scheduling, and repeated random seeds. Training continued until a target training accuracy was reached.
- Architecture: The experiments used the ResNet architecture, specifically ResNet-50 with 50 hidden layers.
- Optimisation: Training used Adam optimisation, dataset-specific initial learning rates, a step scheduler, and batch size 64.
- Evaluation procedure: Models were trained until reaching training accuracy 0.97, with required epochs recorded as a computational-cost measure.
- Implementation: Models used random flips, rotations, and affine transformations for data augmentation.
- Evaluation procedure: Each experiment was replicated across three random seeds to account for stochastic variation in optimisation.
2.4 Exploring the effects of ground truth uncertainty
The study varied volunteer-agreement thresholds across two camera-trap datasets and evaluated model accuracy, species-level performance, difficulty, pre-training, generalisation, and class imbalance.
- Models accepted labels at 100%, 90%, or 66% volunteer agreement thresholds, thereby introducing different levels of ground truth uncertainty.
- Experiments primarily used the Prickly Pear Project Kenya dataset and were replicated for key analyses on Snapshot Serengeti.
- Balanced training datasets were formed by randomly sampling 700 images per species, duplicating some underrepresented species when necessary.
- Models were trained from random initialization or pre-trained on ImageNet, then evaluated using overall accuracy and species-level true positive rate.
- Example difficulty was measured using Shannon entropy of volunteer classifications, with higher values indicating greater disagreement and harder images.
- Class-imbalance experiments removed the per-class image cap, varied agreement thresholds, and quantified imbalance using Shannon entropy of training-label proportions.
3 Results
Higher volunteer disagreement generally reduced accuracy on difficult images, but training with more uncertain labels improved performance there, especially after ImageNet pre-training. The benefit did not consistently transfer across datasets, while class imbalance favored common species and altered error patterns.
- As volunteer disagreement increased, model accuracy decreased, but lower agreement thresholds reduced this decline and improved performance on difficult images.
- ImageNet pre-training universally increased performance and amplified the benefits of training with higher ground truth uncertainty.
- The highest overall accuracy across species was achieved by ImageNet pre-training followed by training with the highest ground truth uncertainty.
- Training under higher ground truth uncertainty improved several species-level accuracies, with effects varying by species and pre-training.
- Additional Snapshot Serengeti pre-training reduced the required training epochs, but produced no consistent test-performance improvement after fine-tuning.
- Unbalanced training slightly improved overall accuracy and favored common classes, while rare-class accuracy generally declined by about ±5-10%.Unbalanced training also made some model errors more closely resemble volunteer mistakes.
4 Discussion
Training with higher ground truth uncertainty generally improved classification, especially for difficult images, while benefits varied by species and did not transfer reliably across datasets. ImageNet pre-training strengthened accuracy gains and reduced training cost, whereas class imbalance traded improved common-species accuracy for poorer rare-species accuracy.
- Higher ground truth uncertainty improved overall accuracy, particularly for test images that were difficult for human volunteers.
- Species-level accuracy generally improved, with clearest gains often occurring for species where volunteer accuracy was relatively poor.Reported examples included helmeted guineafowl, buffalo, hartebeest, impala, and Grant’s gazelle.
- ImageNet pre-training enhanced overall and species-level accuracy and extended higher-uncertainty benefits to relatively easy images.Models trained from scratch mainly improved on intermediate- to high-difficulty images.
- Higher uncertainty improved accuracy on unseen images from the same dataset but did not improve generalisation to a different dataset.Models pre-trained on Snapshot Serengeti and tested on Prickly Pear Project Kenya showed no performance differences across label-agreement thresholds.
- With unbalanced data, higher uncertainty retained an overall benefit, while class imbalance improved common-species accuracy and reduced rare-species accuracy.Unbalanced models also produced directional errors resembling volunteer mistakes, such as rare species being mistaken for common species.
- ImageNet-pre-trained models outperformed Snapshot Serengeti-pre-trained models, while additional pre-training reduced epochs without improving accuracy.Pre-training on both datasets provided no accuracy benefit over ImageNet alone, although it required fewer epochs.
- The choice between balanced and unbalanced training depends on whether accuracy for rare or common classes is more important.The findings also support using moderate label disagreement and general-image pre-training in ecological image workflows with multiple labels.
- Multiple volunteer classifications can provide information about example difficulty and may support more accurate combined human–deep-learning classifications.The paper also identifies differences in human and model mistakes as potentially useful for integration.
ground truth uncertainty: supplementary figures
The supplementary figures show how label-agreement thresholds, class balance, and volunteer predictions are represented across the study’s camera-trap classification analyses.
- Figure S1 compares Snapshot Serengeti test accuracy and species-level accuracy across volunteer-agreement thresholds, with and without ImageNet pre-training.
- Figure S2 expresses each Prickly Project Kenya class’s training-sample count as a proportion of the most abundant class for each agreement threshold.
- Figure S3 organizes model confusion matrices by agreement threshold and balanced versus unbalanced Prickly Pear training sets.
- Figure S4 provides the corresponding volunteer confusion matrix, using true test labels as rows and volunteer predictions as columns.