Source-linked AI summary
Deep learning tools for the measurement of animal behavior in neuroscience
Mackenzie W. Mathis, Alexander Mathis
TL;DR
Animal behavior is complex and difficult to quantify manually at scale, motivating computational methods that preserve meaningful behavioral detail. The paper reviews deep-learning pose-estimation advances and their adaptation to neuroscience, finding that transfer learning enables excellent results from a few hundred annotated images while tools can reach human-level accuracy and online-compatible speeds.
Problem
Animal behavior is complex, context dependent, and labor intensive to measure, creating a need for meaningful behavioral metrics that inform neuroscience.
Method
The paper reviews deep-learning approaches for animal pose estimation, including transfer learning and tools designed for laboratory behavioral quantification.
Results
Deep-learning tools can reach human-labeling accuracy, support online-feedback applications, and achieve excellent results with only a few hundred annotated images.
Takeaways & Limitations
Shared, easy-to-use tools and continued deep-learning advances are expected to expand practical behavioral quantification in real-world neuroscience.
Abstract
from arXiv · showhide
Recent advances in computer vision have made accurate, fast and robust measurement of animal behavior a reality. In the past years powerful tools specifically designed to aid the measurement of behavior have come to fruition. Here we discuss how capturing the postures of animals - pose estimation - has been rapidly advancing with new deep learning methods. While challenges still remain, we envision that the fast-paced development of new deep learning tools will rapidly change the landscape of realizable real-world neuroscience.
Introduction
Behavior provides a crucial window into neural computations and an animal’s umwelt, but measuring its complexity over time is difficult. Pose estimation uses videography to quantify posture and derive kinematics, dynamics, and actions.
- Introduction: Behavior is the primary observable output of neural computations and reflects an animal’s motivations, needs, and survival value.
- Introduction: Manual behavioral measurement is powerful but labor intensive and limited by human senses.
- Introduction: Long behavioral records must be compressed into meaningful metrics that support understanding of the brain and an animal’s umwelt.
- Introduction: Pose estimation measures posture, defined as the geometrical configuration of body parts, using non-invasive videography.
- Introduction: Estimated poses across time can be transformed into kinematics, dynamics, and actions, and posture’s low dimensionality makes these applications computationally tractable.
2D and 3D (human) pose estimation.
Human pose estimation advanced rapidly with deep learning, supported by expanding benchmarks and evaluation metrics. Three-dimensional estimation remains harder because labeled data are more difficult to acquire, while current highest accuracy uses multiple two-dimensional views.
- 2D pose estimation: Since DeepPose applied deep learning to human 2D pose estimation in 2014, new architectures and standardized benchmarks accelerated performance improvements.
- 3D pose estimation: 3D human pose estimation is more challenging than 2D estimation because 3D labeled data are harder to acquire.
- 3D pose estimation: Current highest accuracy for 3D human pose estimation is achieved by reconstructing estimates from multiple 2D views.
- Dense representations: Dense-pose representations capture soft-tissue shape through 3D point clouds or meshes, but state-of-the-art performance requires body-scanning many subjects.
- Dense representations: No animal-specific dense-representation toolboxes are currently geared toward neuroscience applications.
Animal pose estimation
Animal pose-estimation tools adapt deep learning advances to neuroscience’s practical requirements: limited labels, human-level accuracy, speed, robustness, three-dimensional movement, and multi-subject interactions.
- Animal pose estimation: Deep learning tools offer robust performance, relatively fast GPU inference, and efficient implementations without requiring body models or tedious parameter tuning.
- Animal pose estimation: The review identifies five requirements for neuroscience pose-estimation tools: small training sets, accuracy and speed, robustness, 3D estimation, and multi-subject tracking.
- Animal pose estimation: Laboratory experiments cannot feasibly label more than 20,000 frames, motivating methods that learn from small datasets.
- Animal pose estimation: Useful networks must match human-applied labeling accuracy while remaining computationally tractable for fast analysis.
- Animal pose estimation: Networks should remain robust across experimental setups and video compression for long-term storage and reanalysis.
- Animal pose estimation: Efficient 3D pose estimation is valuable for studying motor learning and control, while multi-subject tracking supports social and animal-object interaction experiments.
1. Small training sets for lab-sized experiments.
Animal pose estimation became feasible for laboratory-scale datasets through transfer learning, which uses pretrained image representations to improve generalization and performance with few annotations.
- 1. Small training sets for lab-sized experiments.: A central challenge is creating tailored animal networks that generalize robustly from little training data.
- 1. Small training sets for lab-sized experiments.: Transfer learning adapts a network trained on one task to another, providing pretrained image representations from larger datasets.
- 1. Small training sets for lab-sized experiments.: DeepLabCut was the first tool described here to leverage human pose-estimation advances for animal applications.
- 1. Small training sets for lab-sized experiments.: Transfer learning reduces training time and improves performance over random initialization, especially with smaller datasets.
- 1. Small training sets for lab-sized experiments.: A few hundred annotated images were sufficient for excellent results across locomotion, reaching, trail-tracking, egg-laying, and hunting tasks.
2. Accuracy & speed.
Animal pose-estimation tools increasingly balance human-level accuracy with fast processing, although network architecture creates an accuracy–speed trade-off.
- Accuracy and speed: DeepLabCut reaches human labeling accuracy, while DeepPoseKit reports about three times the accuracy of LEAP and similar accuracy to DeepLabCut.DeepPoseKit also reports about twice faster batch video processing than DeepLabCut and LEAP on small frame sizes.
- Accuracy and speed: Deeper networks generally provide more generalization ability but run more slowly than stacked-hourglass networks.DeepLabCut uses deeper ResNets and matches human labeling accuracy, whereas MobileNetV2 backbones provide slightly lower accuracy with twice the speed.
- Accuracy and speed: GPU-based pose-estimation packages can process several hundred frames per second in offline modes.
- Accuracy and speed: Figure 2 illustrates animal pose-estimation applications, including body-part accuracy, 3D limb estimates, and unsupervised trajectory analysis.
3. Robustness.
Robustness is essential for long neuroscience video experiments because recordings vary across environments, subjects, and compression levels. Transfer learning, augmentation, and active learning improve robustness, but direct out-of-domain testing has been limited.
- Robustness: Long-term neuroscience recordings must tolerate changes in backgrounds, lighting, cameras, animal strains, and video compression.
- Robustness: Transfer learning, data augmentation, and active learning are used to increase robustness and expand training data with failure cases.
- Robustness: Toolboxes had been evaluated mainly on same-distribution train–test splits rather than direct out-of-domain robustness.
- Robustness: Pretraining on ImageNet strongly improves out-of-domain performance, and better ImageNet-performing networks are more robust, although a gap remains.
- Robustness: DeepLabCut remains accurate after more than 1,000X video compression, with less than 1 pixel additional average error.
4. 3D animal pose estimation.
Animal pose-estimation toolboxes support both 2D and 3D key-point detection, using triangulation or multi-camera reconstruction to obtain 3D poses.
- 3D animal pose estimation: Several animal pose-estimation toolboxes explicitly support 2D and 3D key-point detection.
- 3D animal pose estimation: DeepLabCut predicts camera-invariant 2D key points and uses traditional triangulation to extract 3D key points.
- 3D animal pose estimation: Anipose extends DeepLabCut with multi-camera 3D reconstruction using a wider variety of methods.
5. Multi-animal & object tracking.
Multi-animal and object tracking can be approached by tracking individuals before pose estimation or by detecting multiple body parts and linking them across animals.
- Multi-animal and object tracking: Neuroscience experiments often require tracking interactions among multiple animals or between animals and objects.
- Multi-animal and object tracking: One approach applies pose estimation after tracking individuals, while another extracts multiple detections per animal and links them.
- Multi-animal and object tracking: Body-part detections can be linked using part affinity fields, pairwise predictions, geometrical constraints, and combinatorics.
The impact on experimental neuroscience
Deep learning-assisted pose estimation expands neuroscience experiments from constrained tasks to complex, naturalistic, and ethologically relevant behaviors while reducing manual labor. These tools support flexible tracking of multiple animals, objects, and specialized body parts, although occlusion and generalization remain challenges.
- Behavioral space: Pose estimation can support behavioral paradigms spanning trial-based tasks, complex motor actions, and naturalistic behaviors.The behavioral-space framework includes classical conditioning, 3D reaching, looming-stimulus escape, and pup retrieval.
- Motor control & kinematics.: Deep learning tools make detailed kinematic studies more feasible, especially for freely moving, small, aquatic, or unmarked animals.The review expects these studies to leverage large-scale recordings and behavioral monitoring.
- Natural behaviors & ethologically relevant features.: Deep learning has enabled pose-based analysis of natural behavior, including 3D hunting-cheetah tracking and studies of human walking synchronization.The cited human study found pairs tended to walk either in phase or exactly out of phase.
- Natural behaviors & ethologically relevant features.: Flexible feature detection enables tracking of tiny, dexterous, and ethologically relevant body parts such as snouts, whiskers, antennae, and tentacles.The review describes DeepLabCut tracking mouse snouts during odor-trail following and anticipates broader use in natural environments.
- Trial-based behaviors: Behavioral quantification is being applied to decision-making paradigms, including broad movement encoding across the brain and replacement of traditional eye, paw, and lick detection.The International Brain Laboratory has assessed DeepLabCut for replacing conventional monitoring methods.
- Remaining challenges in pose estimation.: Crowded scenes, occlusions, and out-of-domain scenarios remain difficult for pose-estimation systems.Anatomical constraints can improve detection but increase computational complexity and reduce speed.
Outlook & Conclusions
The field is continuing toward more robust, accessible, and broadly applicable animal pose-estimation tools. The review emphasizes specialized networks, open-source resources, and shared tools while recognizing unresolved data and generalization challenges.
- Outlook & Conclusions: Pose-estimation research is pursuing larger balanced datasets, greater use of temporal information, and new algorithms for occlusions and out-of-domain data.These directions are presented as ways the field may continue evolving.
- Outlook & Conclusions: Universal animal pose detectors are difficult to envision because animals have diverse body plans and experimentalists have different needs.The review instead favors specialized networks trained on large datasets for specific animals.
- Outlook & Conclusions: Shared animal-specific networks and open-source code could make analysis more accessible and contribute to reproducible science.The authors describe efforts toward tools that require little or no user training.
- Outlook & Conclusions: Deep learning tools are practical for laboratory use, potentially matching human or marker-based labeling accuracy and supporting closed-loop experiments.The review connects these capabilities to studying links between neural systems and behavior.
Highlighted References:
The highlighted references trace the development of deep learning pose estimation from human systems to animal-focused tools and behavior-analysis pipelines. They emphasize accuracy, speed, transfer learning, tailored tracking, and broader behavioral applications.
- Human pose-estimation foundations: OpenPose enabled real-time multi-person 2D pose estimation by jointly detecting body parts and linking proposals across individuals.The toolbox later expanded to body, hand, facial, foot, and 3D keypoints.
- Human pose-estimation foundations: DensePose established dense correspondences between RGB images and human bodies using a 50K-person dataset and addressed occlusions in the wild.The system was reported to run at up to 25 frames per second.
- Animal pose-estimation tools: Three-D Safari extended animal pose estimation to zebra pose, soft-tissue shape, and texture in visually challenging wild settings.The approach used the SMALST model and targeted zebras blending into safari backgrounds.
- Human pose-estimation foundations: DeeperCut improved multi-person pose estimation using stronger body-part detectors and image-conditioned pairwise terms incorporated through graph cutting.These pairwise terms predict body-part locations from other body-part locations.
- Animal pose-estimation tools: Multi-animal 3D shape and motion recovery combined methods to fit 3D shapes to multiple quadrupeds and released annotations and silhouettes for eleven animal videos.The referenced animals ranged from camels to bears.
- Animal pose-estimation tools: DeepLabCut introduced markerless animal pose estimation of user-defined body parts and achieved nearly human-level performance with 50–200 images.It was benchmarked across flies, hand articulations, and mouse open-field behavior.
- Behavior-analysis toolboxes: DeepBehavior combines OpenPose, YOLO, and Tensorbox for animal and human behavior imaging, with additional MATLAB analysis scripts.Applications include pellet reaching, social behavior, three-chamber tests, and 3D human kinematics.
- Animal pose-estimation tools: LEAP provides a lightweight deep neural network framework for fast animal pose estimation with graphical labeling and training interfaces.The work also applied unsupervised behavioral analysis directly to posture.