Source-linked AI summary
Measuring behavior across scales
Gordon J. Berman
TL;DR
The review addresses how behavior should be quantified across length and time scales, a conceptual challenge sharpened by advances in neural-circuit measurement. It synthesizes theoretical, analytical, computational, and experimental approaches, concluding that behavioral representations must balance abstraction with fidelity while recognizing important methodological limits.
Problem
Behavior lacks a quantitative language that specifies which measurements should describe movements across the multiscale, distributed dynamics of behavior.
Method
The review synthesizes behavioral representations and data-driven methods spanning organismal scales, postural time series, dynamical models, and discrete or continuous behavior.
Results
Behavioral representations can be discrete or continuous, with continuous approaches allowing non-stereotyped dynamics and discrete approaches offering clustering when data support separable groups.
Takeaways & Limitations
Useful behavioral measurements should be parsimonious while balancing consistency, fidelity, interpretability, and scalability according to the scientific question.
Takeaways & Limitations
AR-HMM representations require hand-tuned time scales and assume behavioral-state durations follow exponential distributions, despite durations spanning orders of magnitude.
Abstract
from arXiv · showhide
The need for high-throughput, precise, and meaningful methods for measuring behavior has been amplified by our recent successes in measuring and manipulating neural circuitry. The largest challenges associated with moving in this direction, however, are not technical but are instead conceptual: what numbers should one put on the movements an animal is performing (or not performing)? In this review, I will describe how theoretical and data analytical ideas are interfacing with recently-developed computational and experimental methodologies to answer these questions across a variety of contexts, length scales, and time scales. I will attempt to highlight commonalities between approaches and areas where further advances are necessary to place behavior on the same quantitative footing as other scientific fields.
MEASURING BEHAVIOR ON THE ORGANISMAL SCALE
Quantifying natural behavior requires representations that balance consistency, fidelity, interpretability, and scalability across behavioral scales. Existing approaches trade naturalism, descriptive richness, throughput, reproducibility, and labor.
- Measurement criteria: Naturalistic behavioral measurement seeks consistency, fidelity, interpretability, and scalability from high-quality data with minimal artificial constraints.Fidelity describes systems accurately and completely, while interpretability connects measurements to biological underpinnings and scalability limits manual and computational costs.
- Measurement criteria: Fidelity and interpretability are inherently in tension: removing details makes measurements more understandable but less accurate.The review therefore favors parsimonious descriptions of multiscale behavioral processes rather than maximizing either property alone.
- Existing approaches: Paradigmatic experiments provide high-throughput, consistent measurements but constrain animals to low-dimensional actions within restricted environments.Such designs can tune the animal to the quantification scheme rather than capture its typical repertoire.
- Existing approaches: Coarse non-paradigmatic variables such as mean velocity preserve naturalism and throughput but capture dynamics at only one scale.They may suit sleep-wake studies but miss precise movements such as grooming patterns.
- Existing approaches: Human-defined behavioral classifications provide richer descriptions but require extensive labor, show user-specific variability, and limit reproducibility.They also assume discrete behavioral states without demonstrating from the data that this representation is appropriate.
- Current progress: Automation and supervised machine learning have increased throughput, improved repeatability, and reduced manual effort for several existing measurement approaches.These advances are especially evident in small organisms such as worms, flies, and zebrafish larvae.
STEREOTYPY AS A GENERAL PRINCIPLE
Biolocomotion is comparatively amenable to quantitative analysis because its ethological context, mechanics, separable actions, and physical constraints support shared representations. These properties motivate stereotypy-based, unsupervised approaches for broader behavioral analysis.
- Biolocomotion: Biolocomotion commonly represents behavior with dynamic trajectories of center-of-mass, body-bending, or limb motion.Researchers study how animals move through their environments using a generally agreed-upon measurement framework.
- Biolocomotion: Clear movement goals and Newtonian mechanics provide a mathematical bridge between behavioral scales in biolocomotion.Robots can serve as physical equivalents of generative models when the underlying mathematics is difficult to analyze directly.
- Biolocomotion: Physical constraints and stereotyped actions often reduce locomotion to a small number of movement patterns or gaits.This separability makes biolocomotion particularly suitable for quantitative representations.
- Stereotypy: Data-driven unsupervised methods build on the observation that many animal movements are low-dimensional relative to total movement capacity and recur similarly.Their development requires precise definitions of low-dimensionality, similarity, and movement, while reserving humans for defining stereotypy and interpreting outputs.
- Stereotypy: Across organisms, automated stereotyped-behavior methods share the questions of how to define behavioral similarity and quantify differences.The studies use technically different approaches but converge on these underlying conceptual issues.
FINDING STEREOTYPED MOVEMENTS IN BEHAVIORAL DATA
Automatic identification of stereotyped behavior generally converts video into low-dimensional postural dynamics and then into a behavioral representation. The resulting representation can also reveal temporal patterns and behavioral sequences.
- Pipeline: Most automated analyses first extract a low-dimensional postural time series from video data.Postural measurements reduce the animal’s observed body configuration to a tractable representation before dynamics are analyzed.
- Pipeline: Postural time series are translated into a dynamical representation that supports isolating individual stereotyped actions.This shared framework links posture, movement dynamics, and behavioral identity.
Extracting postural time series
Extracting posture converts high-dimensional video into low-dimensional body-configuration measurements, but the appropriate representation depends strongly on morphology. Direct body-part tracking is interpretable yet remains impractical for very large datasets.
- Postural representation: Posture describes an animal’s body and limb configuration at each time point, preferably in the animal’s body frame.Body-frame measurements separate behavioral description from the animal’s spatial position and orientation.
- Postural representation: Different morphologies require different postural representations: worms can use centerline motion, whereas flies require legs, wings, and other body movements.Rodents and humans require still other representations because of their greater flexibility or distinct movement structure.
- Postural representation: The general objective is to reduce thousands to millions of pixel values to a low-dimensional set of numbers describing posture.This compression makes subsequent behavioral analysis tractable while retaining relevant body configuration information.
- Tracking challenges: Tracking individual joints, leg tips, tails, or heads is interpretable but difficult, often requiring markers or substantial manual correction.For datasets containing up to billions of images, individual body-part tracking is not currently practical, especially in two-dimensional images.
Building representations of dynamical behavior
Behavioral dynamics are represented by transforming postural measurements into time-varying dynamical descriptions that can expose stereotyped actions across multiple time scales. The appropriate representation depends on morphology, available data, and assumptions about temporal structure.
- Building dynamical representations: Stereotyped behavior requires representing trajectories of postural measurements through time rather than isolated postures.Dynamical representations can be built by fitting differential equations or using features that incorporate dynamics.
- Building dynamical representations: Manual dynamical features are simple to implement but may omit unmeasured dynamics and combine variables with incompatible units.Velocity, angular velocity, acceleration, and distance may require conversion factors or assumptions about equal variance.
- Building dynamical representations: C. elegans motion can be largely described by a single phase variable representing a traveling wave along the body.A deterministic dynamical-system fit turns this phase variable into a behavioral representation.
- Building dynamical representations: AR-HMMs model short postural segments with linear dynamical systems while switching among behavioral states on longer time scales.The approach produces dynamical and behavioral representations simultaneously, with the state controlling the postural-output dynamics.
- Building dynamical representations: AR-HMMs require a time-scale parameter for state persistence and assume exponentially distributed behavioral-state durations.This can conflict with behaviors whose durations range from tens of milliseconds to a night’s sleep.
- Building dynamical representations: Motif-finding and wavelet transforms provide multi-scale alternatives by representing repeated patterns or frequency content across varied temporal scales.Motif-based behavioral representations comprise the set, frequency, and ordering of motifs, whereas wavelet representations use amplitudes across frequencies.
Discrete vs. continuous behavioral representations
Behavioral representations may be discrete or continuous, and discrete structure can sometimes emerge from continuous dynamical representations. The preferred choice depends on the data and experimental question, because each form preserves different information and imposes different assumptions.
- Discrete and continuous representations: Discrete representations use clusters or motifs, whereas continuous representations use densities or non-piecewise dynamical models.Discrete states can sometimes be derived from continuous representations through fixed points or density peaks.
- Continuous representations: Continuous representations preserve edge distinctions and allow non-stereotyped dynamics instead of assigning every time point to a cluster.Fruit flies perform non-stereotyped behaviors approximately half of the time.
- Choosing a representation: Representation choices should match the experimental question because nonlinear embeddings distort length scales and single-time-scale models can over- or under-partition data.When clusters are present, clustering in the information-preserving high-dimensional space avoids dependence on a nonlinear embedding’s specific distortions.
FUTURE CHALLENGES
Future behavioral representations should link postural dynamics with space, other behavioral modalities, other individuals, and neural dynamics.
- FUTURE CHALLENGES: Future work aims to connect postural dynamics with spatial variables, other behavioral modalities, other individuals, and neural dynamics.
Joint representation of space and posture
Many behavioral representations remove spatial position and orientation to measure movement in an animal-centered frame, but this prevents a true joint representation of space and posture. Combining these variables also requires arbitrary unit-conversion choices.
- Joint representation of space and posture: Postural representations often eliminate position, orientation, and their derivatives, despite the relevance of location–movement interactions in neuroscience and social behavior.
- Joint representation of space and posture: Adding posture and spatial variables requires unit conversion because their dynamical features and spatial measurements have different units.At least one arbitrarily chosen parameter is required.
- Joint representation of space and posture: Conditioning behavior on position or position on behavior, or averaging response fields across individuals, does not provide a true joint representation.The unresolved comparison is whether repeated identical motions at different locations are closer than different motions at the same location.
Collective and social behavior
Collective and social behavior require measures that capture coordinated motion across individuals as well as non-physical signals exchanged during interactions.
- Collective movement is often summarized with an order parameter related to the proportion of individual velocities pointing in the same direction.
- Social behavior cannot be fully described through limb and body motion alone because animals also produce audio and substrateborne signals.
- General linear models have related behavioral dynamics to audio dynamics by identifying features that predict a song or song type.
Linking neurons to behavior
Linking neural activity to behavior is increasingly necessary as recordings from freely behaving animals expand, but current analyses mainly use correlation or decoding.
- Freely behaving neural recordings make joint representations of neural activity and behavior increasingly necessary.
- Most current neuro-behavioral analyses ask what behavior can be predicted from neural dynamics, or what neural dynamics can be predicted from behavior.
- One proposed avenue combines experimentally tested computational models of neural dynamics with high-resolution behavioral measurements and perturbations.
TOWARD THEORIES OF BEHAVIOR
The review argues that understanding how animals generate movement requires theories and models that connect behavioral measurements to goals, neural function, and future experiments.
- Conceptual challenges remain central because theories and models are needed to contextualize measurements and generate predictions for future experiments.
- Behavioral theories can treat movements in the context of the tasks they help animals perform, including the evolutionary goal of passing genes to subsequent generations.
- Computational ethology seeks “software explanations of behavior” alongside the mapping and manipulation of neural circuits.
- Behavioral observations can support inferences about brain function without detailed models or knowledge of brain dynamics or morphology.
- Stereotyped movements provide a measurement scale grounded in observed movement low-dimensionality and common neural circuitry such as central pattern generators.
- Further theoretical concepts are needed to expand, refine, and apply behavioral measurement methods.