Camera trap classification with deep learning under ground truth uncertainty

1 Introduction

• Supervised image classification requires costly annotation, prompting researchers to use volunteer citizen-science labels and accept images below unanimous agreement thresholds.Lowering the volunteer-agreement threshold retains more disputed examples but makes the assumed ground-truth labels less certain.Source: §1 · p. 4
• Disagreement may expose models to difficult examples and increase training data, but incorrect labels can also teach wrong class associations.Earlier work applying incorrect labels uniformly did not capture their realistic associations with particular classes or difficult images, leaving their overall effects unclear.Source: §1 · p. 5
• Ground-truth uncertainty may interact with generalisation to new contexts and class imbalance, which can favor abundant classes at the expense of rare ones.Because some species may be more prone to misidentification, changing the volunteer-agreement threshold can also change the dataset’s class balance.Source: §1 · pp. 5–6
• The study uses two camera-trap datasets with volunteer and expert labels to investigate how ground-truth uncertainty affects deep-learning classification.Its questions cover difficult images, species-level accuracy, generalisation between datasets, and interaction with class imbalance.Source: §1 · p. 6
• The authors frame their investigation around whether uncertainty improves performance on difficult images, changes species-level accuracy, affects cross-dataset generalisation, or interacts with imbalance.They position the findings as relevant to settings where multiple annotators provide labels.Source: §1 · p. 6

2.1 Datasets

• Prickly Pear Project Kenya contains 186,861 images collected in Laikipia County, Kenya; the images contain >40 mammal species and >60 bird species.Source: §2.1 · p. 6
• The Kenya dataset has substantial class imbalance, reflecting variation in species abundance and detectability.Source: §2.1 · p. 7
• A total of 8,290 Zooniverse volunteers classified the Kenya images, with experience ranging from none to extensive.Source: §2.1 · p. 7
• A relatively large number of images (n = 26,952) were classified by an expert in Zooniverse projects; another cited passage states that events were expert-classified without reporting a count.Source: §2.1 · p. 7
• Snapshot Serengeti contains more than 40 mammal and bird species photographed in Tanzania’s Serengeti National Park.The analyzed subset comprised 848,665 images from 319,915 capture events.Source: §2.1 · p. 7
• The Snapshot Serengeti analysis used the subset with consensus species classifications.Source: §2.1 · p. 7

• The datasets focused on 17 Prickly Pear Project Kenya species and 16 Snapshot Serengeti species, excluding images outside those focal species or with excessive multiple-species classifications.Source: §2.2 · p. 7
• Snapshot Serengeti capture events contained 1–12 images, and the study retained the middle image in each sequence.Events had a median of 3 images and a mean of 2.65.Source: §2.2 · p. 7
• The retained datasets contained 128,996 Kenya images and 143,993 Serengeti images, including both volunteer- and expert-classified subsets.Kenya included 118,736 volunteer-classified and 10,260 expert-classified images; Serengeti included 141,008 volunteer-classified and 2,985 expert-classified images.Source: §2.2 · p. 7
• Images were cropped to remove metadata and logos, then resized to 300 × 160 pixels for Kenya and 300 × 210 pixels for Serengeti.Source: §2.2 · pp. 7–8
• Training labels used majority-vote acceptance thresholds of 100%, 90%, or 66% agreement, while expert-classified images were tested against expert labels.Source: §2.2 · p. 8

2.3 Model architecture and implementation

• Experiments used ResNet-50, a residual network with 50 hidden layers (He et al., 2016).The models were implemented using PyTorch.Source: §2.3 · p. 8
• Models used Adam (Kingma and Ba, 2015) with initial learning rates of 5 × 10−3 from scratch and 2×10−3 for pre-trained models, a step-size-5 scheduler with factor 0.7, and batch size 64.All models were trained until training accuracy reached 0.97, and the number of epochs required was recorded.Source: §2.3 · p. 8
• All models used random horizontal flips (p = 0.5), rotations (±15◦), and affine transformations with up to 10% translation in both directions.Each experiment was replicated across three random seeds to account for stochastic variation in optimization, including initialization and mini-batch sampling.Source: §2.3 · p. 8

2.4.1 Does training under ground truth uncertainty improve performance on difficult images?

• Balanced models trained at 100%, 90%, or 66% label agreement were evaluated on expert-classified test images, with difficulty measured by volunteer disagreement entropy.Models were trained from scratch or with ImageNet pre-training; images were evaluated across equal-width entropy bins, excluding a few high-entropy images whose bins were too small.Source: §2.4.1 · pp. 9–10

2.4.2 How is species-level classification accuracy affected by ground truth uncertainty?

• Species-level performance was measured with true positive rate (recall or sensitivity), and volunteer accuracy was measured with the same metric for comparison.Source: §2.4.2 · p. 10

2.4.3 Does training on more uncertain images improve generalisation between datasets?

• For cross-dataset evaluation, models trained on balanced Snapshot Serengeti data at 100% or 66% agreement were fine-tuned on Prickly Pear Project Kenya at 100%, 90%, or 66% agreement.Models were trained from scratch or ImageNet-pre-trained, fine-tuned to 0.97 training accuracy, and evaluated across Kenya test-set entropy bins.Source: §2.4.3 · p. 10

2.4.4 Does ground truth uncertainty interact with class imbalance?

• To test interaction with class imbalance, ImageNet-pre-trained models used Kenya data without a per-class image cap and 100%, 90%, or 66% agreement thresholds.The thresholds directly changed training-set size and imbalance; balance was quantified with Shannon entropy, and overall and species-level performance were assessed.Source: §2.4.4 · p. 12

3.0.1 Does training under ground truth uncertainty improve performance on difficult images?

• Training under higher ground truth uncertainty improved model performance, with its benefits amplified by ImageNet pretraining; across all species, highest classification accuracy followed ImageNet pretraining and the highest uncertainty.Source: §3.0.1 · p. 12

3.0.4 Does ground truth uncertainty interact with class imbalance?

• With unbalanced data, lower agreement thresholds improved accuracy across entropy classes, especially difficult images, while unbalanced models often did worse on high-entropy images than balanced models.Unbalanced training generally improved accuracy for common classes and reduced it for rare classes.Source: Figure 2 · p. 13, §3.0.4 · p. 17

3.0.3 Does training on more uncertain images improve generalisation between datasets?

• ImageNet followed by Snapshot Serengeti pretraining reduced the epochs needed to reach 0.97 training accuracy, while training from scratch required more than twice as many epochs as other models.Fine-tuning on 100% agreement data required the fewest epochs, with a slight further reduction using 66% Snapshot Serengeti data; lower agreement thresholds generally required more epochs.Source: Figure 3 · p. 14, §3.0.3 · p. 16

3.0.2 How is species-level classification accuracy affected by ground truth uncertainty?

• Species-level gains from higher uncertainty varied, while ImageNet pretraining improved most species and sometimes matched or exceeded volunteer performance.The strongest uncertainty-related gains included helmeted guineafowl, buffalo, and warthog; the largest pretraining gains included dikdik, helmeted guineafowl, and bird (other).Source: §3.0.2 · pp. 13–14, Figure 4 · p. 15

3.0.3 Does training on more uncertain images improve generalisation between datasets?

• Higher uncertainty did not consistently improve cross-dataset test performance; additional Snapshot Serengeti training gave no accuracy gain after ImageNet pretraining.Without ImageNet pretraining, Snapshot Serengeti training generally gave a small accuracy boost; the uncertainty-related effects were small and limited to one entropy bin each.Source: §3.0.3 · p. 16

3.0.4 Does ground truth uncertainty interact with class imbalance?

• Unbalanced training generally improved test accuracy for common species and reduced accuracy for rare species.Source: §3.0.4 · p. 17
• Accuracy was consistently higher for six species and lower for six others, while five species showed no consistent pattern.Accuracy rose for impala, vulturine guineafowl, plains zebra, dikdik, elephant, and baboon; it fell for helmeted guineafowl, buffalo, livestock, vervet monkey, camel, and warthog; Grévy’s zebra, bird (other), giraffe, hare, and hippopotamus varied.Source: §3.0.4 · p. 17
• For plains versus Grévy’s zebra and vulturine versus helmeted guineafowl, unbalanced models and volunteers more often misclassified the rarer species as the common one.Balanced models tended to make errors in both directions at similar rates; unbalanced models and volunteers did not make the reverse errors at similar rates.Source: §3.0.4 · p. 17
• Species-specific accuracy differences between unbalanced and balanced training were generally around ±5–10%, varying by species and agreement threshold.The reported differences showed no clear overall pattern across agreement thresholds.Source: §3.0.4 · p. 17, Figure 5 · p. 18

difficult images

• Higher ground-truth uncertainty improved overall test accuracy, especially on images that were more difficult for volunteers.The benefit was also observed with unbalanced training data.Source: difficult images · p. 19, §4 · p. 19, truth uncertainty are still evident · p. 22
• Species-level accuracy improved for several classes where volunteer accuracy was relatively poor, including helmeted guineafowl and buffalo in Prickly Pear Project Kenya, and hartebeest, impala, and Grant’s gazelle in Snapshot Serengeti.Source: difficult images · p. 19

4.2 The benefits of ground truth uncertainty are enhanced by pre-training

• ImageNet pre-training extended uncertainty’s benefit to relatively easy images and improved accuracy while reducing training epochs; additional Snapshot Serengeti pre-training reduced fine-tuning cost but not accuracy.ImageNet-pre-trained models outperformed those pre-trained on Snapshot Serengeti, despite the latter’s closer similarity to test data.Source: §4.2 · pp. 20–21

contexts

• Models fine-tuned and tested across Snapshot Serengeti and Prickly Pear Project Kenya showed no performance differences across label-agreement thresholds, so greater uncertainty did not improve cross-dataset generalisation.The authors suggest exploring uncertainty in object-detection pipelines, noting that detectors may fail to bound animals in difficult images.Source: contexts · pp. 21–22

truth uncertainty are still evident

• Unbalanced training generally improved accuracy for common species but reduced it for rare species, and shifted confusions toward volunteers’ directional errors from rare to common species.Examples included Grévy’s zebra mistaken for plains zebra and helmeted guineafowl for vulturine guineafowl; whether to include all uncertain examples depends on the priority given to common or rare classes.Source: truth uncertainty are still evident · p. 22
• Combined human–model classifications may be more accurate when their mistakes differ, enabling disagreement to be flagged for validation; similar mistakes may pass undetected.Source: truth uncertainty are still evident · p. 22

4.5 Conclusions and recommendations

• Higher ground-truth uncertainty improved test classification accuracy, especially on volunteer-difficult images, but not generalization to novel contexts; imbalance helped common and hurt rare species without clear interaction with uncertainty.Source: §4.5 · p. 23
• Practitioners can maximize accuracy with moderate label disagreement and ImageNet-pretrained models; pretraining reduces training epochs, while balanced versus unbalanced data should depend on rare- versus common-class priorities.Source: §4.5 · p. 23
• Multiple volunteers can help identify difficult examples, and distinct human and model error patterns could be leveraged to improve combined classifications.Source: §4.5 · p. 23
• Human classifiers can identify behaviors and species interactions beyond a model’s task, while ecological research participation can support wellbeing, skill development, and connection to nature.Source: §4.5 · p. 23