PaperlayerView exact revision on arXiv

Source-linked AI summary

Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning

Mohammed Sadegh Norouzzadeh, Anh Nguyen, Margaret Kosmala, Ali Swanson, Meredith Palmer, Craig Packer, Jeff Clune

arXiv:1703.05830v5cs.CVcs.LG

TL;DR

Camera-trap images offer abundant wildlife data, but extracting species, counts, and behaviors remains labor-intensive. The paper applies deep neural networks to 3.2 million Snapshot Serengeti images spanning 48 species. With confidence thresholding, the system automates 99.3% of animal identification at 96.6% accuracy, matching human volunteers while saving over 17,000 hours of labeling effort.

Abstract

from arXiv · show

Having accurate, detailed, and up-to-date information about the location and behavior of animals in the wild would revolutionize our ability to study and conserve ecosystems. We investigate the ability to automatically, accurately, and inexpensively collect such data, which could transform many fields of biology, ecology, and zoology into "big data" sciences. Motion sensor "camera traps" enable collecting wildlife pictures inexpensively, unobtrusively, and frequently. However, extracting information from these pictures remains an expensive, time-consuming, manual task. We demonstrate that such information can be automatically extracted by deep learning, a cutting-edge type of artificial intelligence. We train deep convolutional neural networks to identify, count, and describe the behaviors of 48 species in the 3.2-million-image Snapshot Serengeti dataset. Our deep neural networks automatically identify animals with over 93.8% accuracy, and we expect that number to improve rapidly in years to come. More importantly, if our system classifies only images it is confident about, our system can automate animal identification for 99.3% of the data while still performing at the same 96.6% accuracy as that of crowdsourced teams of human volunteers, saving more than 8.4 years (at 40 hours per week) of human labeling effort (i.e. over 17,000 hours) on this 3.2-million-image dataset. Those efficiency gains immediately highlight the importance of using deep neural networks to automate data extraction from camera-trap images. Our results suggest that this technology could enable the inexpensive, unobtrusive, high-volume, and even real-time collection of a wealth of information about vast numbers of animals in the wild.

Background and Related Work

Deep learning uses supervised deep neural networks to learn image features automatically, supporting computer-vision approaches to wildlife-image analysis at much larger scale than earlier methods. The Snapshot Serengeti dataset provides 3.2 million images from 48 species, but event-level labels can introduce noise and the data are highly imbalanced.

Deep Learning.

  • Deep neural networks learn multiple abstraction levels automatically, transforming raw pixels into features such as edges, textures, and object parts before prediction.Supervised learning uses paired images and labels, while convolutional layers extract information from overlapping regions and softmax outputs class probabilities.Source: Figure 3 · p. 2, Machine Learning · p. 2, Deep Learning · p. 2

Related Work.

  • 92.0% accuracy was reported for this work’s best network versus approximately 57% for Gomez et al.’s best network on species identification.Unlike Gomez et al., this study retains all 48 classes and also addresses counting, behavior, and presence of young.Source: Related Work · p. 3

Snapshot Serengeti Project.

  • 3.2 million images from 48 species make Snapshot Serengeti a large-scale testbed for automated wildlife-image analysis.The dataset contains 1.2 million capture events collected by 225 camera traps; volunteers supplied consensus labels after classifying species, counts, behaviors, and young.Source: Snapshot Serengeti Project · p. 3
  • 75% of capture events are empty, and severe class imbalance can yield high overall accuracy while leaving rare species poorly recognized.Experiments found extremely low accuracy on rare classes, with some classes near 0%; labels assigned from capture events to individual images can also add training noise.Source: Snapshot Serengeti Project · p. 3

Experiments and Results

The experiments evaluate a two-stage, multitask deep-learning pipeline for animal detection, species identification, counting, and attribute prediction. Across these tasks, ensembles achieve strong performance, while capture-event aggregation improves accuracy by about 1%.

Pipeline design

  • A two-stage pipeline first detects animal presence, then extracts species, counts, and additional attributes; 75% empty images make automating detection especially consequential.The information-extraction stage includes species identification, animal counting, and behavior or young-presence prediction.Source: Experiments and Results · p. 3

Capture-event evaluation

  • Capture-event prediction is about 1% more accurate than individual-image prediction because averaging across event images mitigates noisy event-level labels.Event-level labels are assigned to every image in an event, so empty images can receive animal labels during training.Source: Figure 4 · p. 4

Experimental setup

Task II: Identifying Species

  • 94.9% top-1 and 99.1% top-5 accuracy were achieved by the ensemble for species identification on the expert-labeled test set.The best single model, ResNet-152, reached 93.8% top-1 and 98.8% top-5 accuracy; Table 1 compares architectures.Source: Task II: Identifying Species · pp. 4–5, Table 1 · p. 5

Task III: Counting Animals

  • 63.1% top-1 accuracy and 84.7% within +/- 1 bin were achieved by the ensemble for animal counting on the expert-labeled test set.Counting is formulated as classification into 12 animal-number bins because bounding-box labels are unavailable.Source: Task III: Counting Animals · p. 5

Task IV: Additional Attributes

  • 76.2% accuracy, 86.1% precision, and 81.1% recall were achieved by the ensemble across the six additional attributes.The attributes are modeled as a multi-label classification problem with separate two-neuron outputs for each behavior or young-presence label.Source: Task IV: Additional Attributes · p. 5

Saving Human Labor via Confidence Thresholding

Confidence thresholding lets the system match volunteer accuracy while automatically processing most images, saving substantial labeling labor.

Saving Human Labor via Confidence Thresholding

  • At 96.6% volunteer-level classification accuracy, confidence thresholding determines which predictions the system accepts automatically.The pipeline uses a two-stage empty-versus-animal classifier followed by an ensemble model, accepting predictions only above confidence thresholds.Source: Saving Human Labor via Confidence… · pp. 5–6
  • The ensemble reaches 94.9% top-1 species accuracy and 63.1% top-1 counting accuracy on the expert-labeled test set.It also achieves 99.1% top-5 species accuracy and places 84.7% of count predictions within +/-1 bin.Source: Figure 5 · p. 6
  • 99.3% of the 3.2 million images can be automatically identified at 96.6% accuracy, saving 8.4 years of human labeling effort.The system processes empty images automatically and thresholds species predictions on non-empty images.Source: Saving Human Labor via Confidence… · p. 6
  • For counting, the system automatically processes 86.1% of all images while matching the volunteers’ 90.0% accuracy.A 79% confidence threshold allows automatic counting for 44.55% of non-empty images, which comprise 25% of the dataset.Source: Saving Human Labor via Confidence… · p. 6

Discussion and Future Work

The authors identify hybrid human–neural workflows and transfer learning as future directions, while emphasizing that their practical benefits require further verification.

Discussion and Future Work

  • Actual time savings and accuracy effects of hybridizing neural networks with human volunteers still need to be quantified.Potential gains include filtering empty images, accepting high-confidence predictions, and presenting ranked model suggestions.Source: Discussion and Future Work · p. 6
  • Human accuracy could decrease by overaccepting incorrect neural-network suggestions or improve when suggestions surface information humans might miss.The direction of this effect remains an empirical question for hybrid systems.Source: Discussion and Future Work · p. 6
  • Transfer learning could extend automation to camera-trap projects that lack large labeled datasets, but experiments are needed to verify its effectiveness.The proposed strategy is to repurpose knowledge learned from a large labeled dataset for smaller projects.Source: Discussion and Future Work · pp. 6–7

Conclusions

Deep neural networks perform well on the large Snapshot Serengeti dataset, and confidence-based automation can sharply reduce manual labeling while preserving volunteer-level identification accuracy.

Conclusions

  • Deep neural networks perform well on Snapshot Serengeti, although performance is worse for rare classes.The study evaluates state-of-the-art computer-vision methods on the largest existing labeled dataset of wild animals.Source: Conclusions · p. 7
  • 99.3% of animal-identification data can be automated at the volunteers’ 96.6% accuracy, saving over 17,000 hours of manual labor.The authors state that this labor can be redirected to other scientific purposes and can make extraction feasible for projects with fewer volunteers.Source: Conclusions · p. 7

Supplementary Information

The supplementary methods describe image preprocessing and training details, including standardized resizing, augmentation, and stochastic-gradient optimization.

Supplementary Information

  • Original 2,048×1,536 images are scaled to 256×256 pixels before neural-network processing.The resizing reduces computational cost but may slightly distort images because aspect ratios are not preserved.Source: Pre-processing · p. 8
  • Random cropping, horizontal flipping, brightness changes, and contrast changes expose the network to varied images during training.These augmentations are intended to improve resistance to small changes and improve accuracy.Source: Data Augmentation · p. 8
  • A camera-trap image is downsampled and normalized before being input to the neural network.The supplementary example contrasts the original Snapshot Serengeti image with its network input.Source: Figure S.1 · p. 8
  • Networks are trained with backpropagation using stochastic gradient descent with momentum and weight decay.The experiments use Torch and TensorFlow, with hyperparameters and learning-rate policies documented in supplementary tables.Source: Training · p. 8

One-stage Identification

The paper compares a one-stage model that treats empty images as an identification class with the main two-stage pipeline. The one-stage approach reduces model size but faces severe class imbalance, cannot readily reuse the empty-versus-animal module, and performs slightly worse.

One-stage Identification

  • About 75% empty images create substantial class imbalance when empty images and species are modeled in one identification task.The imbalance makes the one-stage problem harder for machine-learning algorithms.Source: One-stage Identification · p. 8
  • The one-stage design cannot reuse an empty-versus-animal module for other similar datasets and produces slightly worse results.Its smaller total model size is therefore accompanied by modularity and performance drawbacks.Source: One-stage Identification · p. 8
  • 220,000 empty images were sampled to match the number of images in the most frequent class, wildebeest, for the one-stage experiment.Four architectures were evaluated using total accuracy, empty-versus-animal accuracy, and species-identification accuracy.Source: One-stage Identification · p. 8
  • 96.3% versus 94.9% accuracy: the two-stage pipeline outperforms the one-step ResNet-50 model on detecting images containing animals.The one-stage result is reported in Table S.3.Source: Table S.3 · p. 8
  • 93.6% versus 90.6% accuracy: the two-stage pipeline outperforms the one-step ResNet-50 model on species identification.This comparison is also reported in Table S.3.Source: Table S.3 · p. 8

Results on the Volunteer-Labeled Test Set

On the volunteer-labeled test set, the models performed strongly across species identification, animal counting, and additional attributes, with similar results across architectures. Comparisons with prior work used a reduced SS-26 dataset and showed higher accuracy than the earlier reported results, while the reasons for that difference remain unclear.

Results on the Volunteer-Labeled Test Set

  • At least 71.3% accuracy, 82.1% precision, and 77.3% recall were achieved for additional-attribute predictions.The ensemble performed best on the description task by a small margin, while architectures were otherwise similar across tasks.Source: Results on the Volunteer-Labeled Test… · p. 9
  • Training from scratch on SS-26 produced substantially higher accuracy than Gomez et al.'s results, with the best network reaching 92.0% top-1 accuracy.SS-26 contains all images from the 26 classes used for comparison because the earlier dataset was not publicly available.Source: Comparing to Gomez et al. 2016 · p. 9
  • The comparison to Gomez et al. is limited because their 88.9% result used about 33,000 manually cropped foreground-animal images rather than the full SS dataset.The paper seeks performance on the full dataset without manual intervention.Source: Comparing to Gomez et al. 2016 · p. 9

Transfer Learning

The paper evaluates ImageNet transfer learning by initializing AlexNet and ResNet-152 before training on Snapshot Serengeti. Transfer learning does not improve accuracy in these experiments, although different hyperparameters might produce different results.

Transfer Learning

  • The transfer-learning models used ImageNet-pretrained AlexNet and ResNet-152 weights as their starting parameters for Snapshot Serengeti training.The static and dynamic hyperparameters matched those of the original experiment.Source: Transfer Learning · p. 9
  • Transfer learning did not increase accuracy when compared with the main experiments.The authors state that ImageNet transfer learning was ineffective in these runs, while leaving open the possibility that different learning rates or other hyperparameters could help.Source: Transfer Learning · p. 9
  • 93.0% top-1 and 98.7% top-5 accuracy were obtained by ResNet-152 for species identification after transfer learning.AlexNet obtained 92.4% top-1 and 98.8% top-5 accuracy on the same task.Source: Transfer Learning · p. 9
  • 62.4% top-1 counting accuracy and 82.6% predictions within one bin were obtained by ResNet-152 after transfer learning.AlexNet reached 59.1% top-1 accuracy and 80.7% within one bin.Source: Transfer Learning · p. 9

Classifying Capture Events

The paper compares aggregating single-image predictions within capture events with directly classifying multiple images together. Aggregation improves capture-event accuracy, whereas single-image training remains effective and avoids several practical complications of multi-image inputs.

Classifying Capture Events

  • The dataset provides capture-event labels, but the neural networks are trained to classify individual images before predictions are aggregated to events.The paper also tests direct event-level classification as an alternative.Source: Classifying Capture Events · p. 9
  • Capture-event accuracy was on average 1% higher than individual-image accuracy when predictions from images in each event were averaged.The authors attribute this likely gain to reduced noise from combining multiple images.Source: Classifying Capture Events · p. 9
  • Training on individual images produced more accurate results than directly inputting all images from a capture event together.The direct multi-image experiment used only three-image events and ignored 55,000 other events for computational reasons.Source: Classifying Capture Events · p. 9
  • Single-image classification avoids variable event sizes, fewer labeled training examples, larger networks, and choices about combining images for feedforward networks.The paper identifies improving use of multi-image event information as a promising direction for future research.Source: Classifying Capture Events · p. 9

Confidence Thresholding

Confidence scores let the system process only images it considers reliable while routing the rest to human labelers, trading coverage for accuracy.

Confidence Thresholding

  • Confidence thresholds control how much data the network classifies automatically and how much is sent to humans for labeling.The authors vary thresholds to construct systems that process different data fractions according to prediction confidence.Source: Confidence Thresholding · p. 9

Improving Accuracy for Rare Classes

Species imbalance makes high overall accuracy misleading because rare classes can be neglected. The authors test weighted loss, oversampling, and emphasis sampling, finding selective gains but no universal solution.

Problem

  • 63% accuracy is possible by predicting only wildebeests, zebras, and Thomson’s gazelle while ignoring the remaining 94% of classes.The dataset is heavily imbalanced, so frequent-class performance can conceal very poor rare-class accuracy.Source: Improving Accuracy for Rare Classes · p. 11

Weighted Loss

  • 40% and 35% accuracy improvements occur for genet and aardwolf after applying the tested imbalance remedy, while most other classes retain their accuracy.The reported gains concern rare classes and do not imply uniform improvement across the dataset.Source: Weighted Loss · p. 11
  • Weighted loss slightly hurts top-1 accuracy but improves top-5 accuracy, suggesting a useful trade-off for imbalanced classification.The method assigns greater cost to errors on rare classes and lower cost to errors on frequent classes.Source: Weighted Loss · p. 11

Oversampling

  • Oversampling slightly hurts overall performance more than the baseline, weighted-loss, and emphasis-sampling methods.The authors state that further investigation is needed to explain this result.Source: Oversampling · p. 12

Emphasis Sampling

  • Emphasis sampling improves accuracy for some rare classes, including civet and rhinoceros by about 40%, and improves dataset top-5 accuracy.It repeatedly presents examples the network misclassifies, giving difficult or rare examples additional training exposure.Source: Emphasis Sampling · p. 12

Overall

  • All three imbalance methods improve some rare classes, but none improves accuracy for every rare class.The authors conclude that additional research is required to improve these methods further.Source: overall · p. 12

Related papers

Loading 1703.05830v5…