PaperlayerView exact revision on arXiv

Source-linked AI summary

Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understanding

Xun Long Ng, Kian Eng Ong, Qichen Zheng, Yun Ni, Si Yong Yeo, Jun Liu

arXiv:2204.08129v2cs.CV

TL;DR

Existing animal behavior datasets provide limited animal, task, sample, environmental, and viewpoint diversity, restricting comprehensive analysis of natural behaviors. The paper introduces Animal Kingdom, a large multi-task dataset, and CARe, which combines general and specific features for recognizing actions in unseen animal types. The dataset and model achieve promising behavior-analysis results, while the basic feature-elaboration design remains limited beyond trained animal types.

  • Problem

    Existing animal behavior datasets have limited animal classes, data samples, tasks, environmental variation, and viewpoints.

  • Method

    The paper constructs Animal Kingdom for video grounding, action recognition, and pose estimation, and proposes CARe using general and specific features for unseen animal types.

  • Results

    CARe achieves the best results in its experiments, while Animal Kingdom covers over 850 animal species across video grounding, action recognition, and pose estimation.

  • Takeaways & Limitations

    Animal Kingdom offers a diverse benchmark for advancing video grounding, animal action recognition, and pose estimation under varied wildlife conditions.

  • Takeaways & Limitations

    The basic CARe model lacks specific feature elaborators for animal types beyond the K types used in training.

Abstract

from arXiv · show

Understanding animals' behaviors is significant for a wide range of applications. However, existing animal behavior datasets have limitations in multiple aspects, including limited numbers of animal classes, data samples and provided tasks, and also limited variations in environmental conditions and viewpoints. To address these limitations, we create a large and diverse dataset, Animal Kingdom, that provides multiple annotated tasks to enable a more thorough understanding of natural animal behaviors. The wild animal footages used in our dataset record different times of the day in extensive range of environments containing variations in backgrounds, viewpoints, illumination and weather conditions. More specifically, our dataset contains 50 hours of annotated videos to localize relevant animal behavior segments in long videos for the video grounding task, 30K video sequences for the fine-grained multi-label action recognition task, and 33K frames for the pose estimation task, which correspond to a diverse range of animals with 850 species across 6 major animal classes. Such a challenging and comprehensive dataset shall be able to facilitate the community to develop, adapt, and evaluate various types of advanced methods for animal behavior analysis. Moreover, we propose a Collaborative Action Recognition (CARe) model that learns general and specific features for action recognition with unseen new animals. This method achieves promising performance in our experiments. Our dataset can be found at https://sutdcv.github.io/Animal-Kingdom.

1. Introduction

Animal behavior analysis supports conservation, wildlife management, and animal health research, but studying diverse, rapidly changing behaviors in long wildlife videos is difficult. Animal Kingdom addresses dataset and task limitations with a diverse benchmark spanning video grounding, action recognition, and pose estimation.

  • Animal behavior analysis supports conservation, wildlife management, and understanding animals’ health and needs.
  • Video monitoring enables continuous observation without physical markers that may alter animals’ natural behaviors.
  • Rapidly changing actions, irrelevant footage, and large wildlife video volumes make behavior analysis laborious and time-intensive.
  • Existing datasets are limited in size, annotation breadth, animal diversity, and representation of varied animal anatomies.
  • Animal Kingdom provides video grounding, action recognition, and pose estimation for analyzing complex, noisy behaviors in the wild.

2. Related Work

Prior animal behavior datasets are often small, narrowly focused, and constrained to particular animals, environments, or tasks. Animal Kingdom broadens coverage across tasks, animal classes, annotations, viewpoints, and environmental conditions to provide a more comprehensive benchmark.

  • Many existing datasets target specific animals, environments, or limited action categories, often using laboratory or industry settings.
  • Prior wildlife datasets commonly focus on few animal classes or a single task, such as mammal pose estimation.
  • Animal Kingdom combines larger task coverage with multi-label actions, diverse animal classes, viewpoints, backgrounds, weather, illumination, and time-of-day conditions.
  • These properties make Animal Kingdom a challenging benchmark for developing and testing animal behavior models.
  • Animal action recognition methods are largely adapted from human action recognition approaches.

3. The Proposed Animal Kingdom Dataset

Animal Kingdom is a large, diverse benchmark spanning video grounding, fine-grained action recognition, and pose estimation across animals, environments, viewpoints, and weather conditions. Its annotations and task design capture practical challenges including intra-class variation, multi-label actions, and long-tailed action distributions.

  • Dataset scale and diversity: 50 hours of long videos, 50 hours of action-recognition clips, and 33K annotated frames support video grounding, action recognition, and pose estimation.
  • Dataset scale and diversity: The dataset covers over 850 species across 6 major animal classes and includes 140 fine-grained action classes.
  • Dataset scale and diversity: Animal Kingdom captures diverse environments, illumination, weather, viewpoints, equipment, and footage types from animals living in the wild.
  • Dataset challenges: Its clips include single- and multi-animal scenes with fine-grained multi-label actions, producing a long-tailed action distribution.
  • Dataset construction: Annotations were manually produced by 23 contributors, including biology experts, with three rounds of quality checks.
  • Dataset challenges: The dataset addresses action variation across animal classes, simultaneous actions, multiple animals per frame, and uneven action frequencies.
  • Dataset tasks and annotations: Pose estimation predicts animal joints using 23 keypoints organized across five animal classes, while action recognition maps video clips to fine-grained labels.

4. The Proposed CARe Model

CARe combines general and animal-specific features to recognize actions across observed and unseen animal types. It approximates unseen-animal-specific features by weighting outputs from existing animal-specific elaborators, then fuses them with general features for classification.

  • CARe combines general and specific features to recognize the same actions for unseen animal types.The model is designed for action recognition beyond the animal types used during training.
  • Base model: The base model extracts a shared general feature and an animal-type-specific feature before classification.A backbone extractor produces base features, which are processed by shared and type-specific elaborators and then passed to the classifier.
  • Base model: During training, all animal types contribute to the general elaborator, while each specific elaborator receives samples from its corresponding animal type.This separates broadly generalizable features from features specialized for particular animal types.
  • Unseen-animal recognition: For an unseen animal type, CARe estimates relevance to the K observed types and combines their specific elaborator outputs.The relevance evaluator processes the sample and intermediate features to produce similarity scores used for collaboration among existing elaborators.
  • Unseen-animal recognition: Spatially aware weighting allows different feature-map positions to use different similarities when approximating the unseen animal’s specific feature.The weights can be tensors over spatial positions rather than a single scalar.
  • Training for generalization: Meta-learning trains the relevance evaluator by simulating an unseen animal type through virtual meta-train and meta-test splits.Alternating base-model training with MLDG training of the evaluator is intended to improve generalization to unseen animals.

5. Experiments

The experiments evaluate adapted state-of-the-art models across action recognition, video grounding, and pose estimation, including long-tailed, unseen-animal, and cross-class settings. Results show gains from long-tailed methods, strong CARe performance for unseen animals, and persistent difficulty with unseen classes and certain animal groups.

  • Experimental setup: The study evaluates I3D, X3D, and SlowFast for action recognition; VSLNet and LGI for video grounding; and HRNet-based methods for pose estimation.The models use their original codes adapted to Animal Kingdom.
  • Action recognition: Focal Loss, LDAM-DRW, and EQL are tested with action-recognition models to address the dataset’s long-tailed multi-label distribution.Evaluation reports overall and head-, middle-, and tail-segment mAP.
  • Action recognition: Higher mAP across head, middle, and tail segments is achieved by action-recognition methods using long-tailed-distribution approaches, such as Focal Loss, than by baselines.The authors identify opportunities for methods addressing long-tailed and multi-label problems.
  • Unseen-animal recognition: CARe achieves the best results for recognizing actions of unseen animal types by combining general features with spatially aware approximated specific features.The unseen-animal experiment uses six shared action classes and separates training animals from test animals.
  • Video grounding: Video grounding is evaluated with mean IoU and Recall@1 at IoU thresholds 0.1, 0.3, 0.5, and 0.7.Recall@1 measures the percentage of queries with at least one correct result among the top returned results.
  • Pose estimation: Pose estimation uses three protocols covering shared species, unseen animal subclasses, and separate major-animal-class evaluation.Protocol 2 reserves 12 subclasses for testing, while Protocol 3 evaluates mammals, amphibians, reptiles, birds, and fishes separately.
  • Pose estimation: Protocol 2 performs comparatively worse than the other pose-estimation protocols, while reptiles and amphibians also obtain lower results in Protocol 3.The paper associates these outcomes with challenges in generalizing poses and estimating varied, camouflaged keypoints.

6. Conclusion

The paper presents Animal Kingdom as a challenging, diverse dataset spanning video grounding, action recognition, and pose estimation, alongside CARe for unseen-animal action recognition. Its variation exposes unresolved challenges and motivates further research in animal behavior analysis.

  • Conclusion: Animal Kingdom covers over 850 animal species across video grounding, action recognition, and pose estimation.The dataset’s within- and between-class variation is presented as a central property.
  • Conclusion: CARe is introduced to improve action recognition for unseen new animal types.The conclusion identifies this as a principal model contribution.
  • Conclusion: Variation across animal classes highlights challenges and reinforces the need for further research in video grounding, action recognition, and pose estimation.The authors expect the work to inspire further animal behavior analysis.

Understanding (Supplementary Materials)

The supplementary materials identify the paper’s authors and their institutional affiliation. They also provide contact information for the research team.

  • Affiliation: The authors are affiliated with Information Systems Technology and Design at the Singapore University of Technology and Design, Singapore.
  • Contact: The supplementary materials list email addresses for the authors using institutional and university domains.

1. Details of Data Collection and Verification

Animal Kingdom was assembled through manual annotation of wildlife videos and extracted frames, with repeated quality checks. The process involved a 23-member team that included biology experts with biodiversity knowledge.

  • Annotation team: The annotation team consisted of 23 members, including biology experts with knowledge of biodiversity.
  • Video annotation: Annotators manually labeled animal and action descriptions framewise across more than 50 hours of YouTube videos.The videos were processed into 30,100 clips, with actions described using defined ethological terms.
  • Quality verification: Three rounds of cross-checking were used to verify video annotations and reduce discrepancies in action terminology.The process reconciled synonymous animal actions and redefined ambiguous movement terms.
  • Pose annotation: For pose estimation, 33K extracted images were labeled with Label Studio and checked through three quality-control rounds.Wild backgrounds, illumination, and weather conditions made keypoint annotation challenging.

2. Diverse Range of Animals

Animal Kingdom spans over 850 animal species organized into six major classes, with class-specific labeling choices for action recognition and pose estimation. The dataset includes visual examples across these diverse animal categories.

  • Diverse Range of Animals: Over 850 animal species are grouped into 6 major animal classes and further divided into subclasses.The classes are organized using appearance, limb number, and movement characteristics.
  • Diverse Range of Animals: Action recognition covers mammals, reptiles, amphibians, birds, fishes, and insects.These six classes receive animal action labels.
  • Diverse Range of Animals: Pose estimation covers 5 major classes: mammals, reptiles, amphibians, birds, and fishes.Insects are excluded because their anatomical differences make pose annotation especially challenging.
  • Diverse Range of Animals: Figures 1 and 2 show the distribution and visual examples of animals across the six main classes.The figures illustrate the breadth of animal categories represented in the dataset.

3. Diverse Range of Actions

Animal Kingdom provides a diverse action vocabulary spanning 140 animal actions and behaviors. These labels cover movement, biological needs, maintenance, communication, defense, and social interactions.

  • Diverse Range of Actions: The dataset contains a diverse range of 140 animal actions.The action collection is designed to encompass varied animal behaviors.
  • Diverse Range of Actions: Its labels include movement, transport, feeding, sensing, resting, and maintenance behaviors.Examples include swimming, carrying, eating, exploring, sleeping, grooming, and washing.
  • Diverse Range of Actions: The action vocabulary also includes communication, aggressive, defensive, social, affection, and sexual behaviors.Examples include chirping, attacking, retreating, playing, hugging, and sexual display.

4. More Examples

The paper illustrates Animal Kingdom’s three annotated tasks through examples of temporal grounding, animal actions, and poses. The figures also highlight variation across animal classes, environments, illumination, and weather.

  • More Examples: Video grounding requires detecting the time sequence corresponding to a language description.Figure 3 provides examples of this temporal localization task.
  • More Examples: Action-recognition examples show shared actions across animal classes and varied actions within the dataset.Figure 4 organizes examples across rows to contrast cross-class and within-dataset action variation.
  • More Examples: The dataset examples include diverse environments, viewpoints, illumination, and weather conditions.Figure 4 includes samples illustrating environmental and visual variation.
  • More Examples: Pose-estimation examples and the distribution of 33K pose annotations are presented in Figures 5 and 6.Figure 5 shows animal poses, while Figure 6 reports their distribution across five major animal classes.

5. Details of the Proposed CARe Model

CARe combines a backbone, general and animal-specific feature elaborators, a classifier, and a relevance evaluator to recognize actions for unseen animals. Its training procedure alternates updates to feature components and relevance estimation using meta-train and meta-test animal types.

  • Model Architecture: CARe consists of a backbone feature extractor, general and specific feature elaborators, and a classifier.The classifier maps concatenated general and specific features to likelihoods for M possible actions.
  • Model Architecture: The backbone adopts the early I3D architecture and processes 16-frame RGB inputs of size 180 × 320.It includes three 3D convolutional layers, two max-pooling layers, and an inception submodule.
  • Unseen-Animal Recognition: The relevance evaluator transforms the base feature and combines it with each specific feature to compute similarity scores for unseen animals.Separate relevance layers produce scores used to collaborate among observed animal-specific elaborators.
  • Training Procedure: The model uses pretrained I3D initialization for the backbone, separate learning rates, SGD optimization, and a 40-epoch schedule.Initial learning rates are used for 30 epochs, followed by rates reduced to 10% for 10 additional epochs.
  • Training Procedure: Training alternates updates to the backbone and elaborators with relevance-evaluator updates using meta-train and meta-test animal types.One animal type is sampled for meta-testing while the remaining K-1 types form the meta-training set.
Loading 2204.08129v2…