PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-camera Realtime 3D Tracking of Multiple Flying Animals

Andrew D. Straw, Kristin Branson, Titus R. Neumann, Michael H. Dickinson

arXiv:1001.4297v1cs.CV

TL;DR

Freely flying animals need to be tracked automatically and in realtime to support large-scale behavioral measurement and experiments that manipulate sensory feedback. The paper presents flydra, a multi-camera system combining Extended Kalman Filtering with Nearest Neighbor Standard Filter data association. It operates with less than 40 msec latency, and low-contrast Drosophila flight is faster and more variable than flight at high contrast.

  • Problem

    Existing flight studies depended on tracking, motivating an automated system that gathers large quantities of accurate data and supports realtime sensory-feedback experiments.

  • Method

    Flydra uses multiple inexpensive cameras, background-subtracted image features, an Extended Kalman Filter, and Nearest Neighbor Standard Filter data association for markerless multi-target tracking.

  • Results

    39 msec median 3D reconstruction timestamp; at low contrasts below 0.16 Michelson contrast, Drosophila flight speed is faster and more variable.

  • Takeaways & Limitations

    The realtime system supports experiments that manipulate sensory feedback and helps quantify freely flying animals’ behavior for neurobiological study.

  • Takeaways & Limitations

    Near-independent target modeling reduces tracking accuracy during periods of near contact.

Abstract

from arXiv · show

Automated tracking of animal movement allows analyses that would not otherwise be possible by providing great quantities of data. The additional capability of tracking in realtime - with minimal latency - opens up the experimental possibility of manipulating sensory feedback, thus allowing detailed explorations of the neural basis for control of behavior. Here we describe a new system capable of tracking the position and body orientation of animals such as flies and birds. The system operates with less than 40 msec latency and can track multiple animals simultaneously. To achieve these results, a multi target tracking algorithm was developed based on the Extended Kalman Filter and the Nearest Neighbor Standard Filter data association algorithm. In one implementation, an eleven camera system is capable of tracking three flies simultaneously at 60 frames per second using a gigabit network of nine standard Intel Pentium 4 and Core 2 Duo computers. This manuscript presents the rationale and details of the algorithms employed and shows three implementations of the system. An experiment was performed using the tracking system to measure the effect of visual contrast on the flight speed of Drosophila melanogaster. At low contrasts, speed is more variable and faster on average than at high contrasts. Thus, the system is already a useful tool to study the neurobiology and behavior of freely flying animals. If combined with other techniques, such as `virtual reality'-type computer graphics or genetic manipulation, the tracking system would offer a powerful new way to investigate the biology of flying animals.

1 Introduction

The paper introduces flydra, an automated multi-camera system for markerless realtime tracking of multiple flying animals. It combines Bayesian state estimation, an Extended Kalman Filter, and Nearest Neighbor Standard Filter data association to support large-scale behavioral experiments.

  • 1 Introduction: The system is designed to collect large quantities of accurate flight data automatically and flexibly.Realtime trajectories can support dynamic sensory-feedback experiments and data-intensive behavioral modeling.
  • 1 Introduction: Flydra uses arbitrary numbers of inexpensive cameras for markerless, realtime tracking of multiple targets.Multiple cameras provide high spatial and temporal resolution over large tracking volumes while reducing occlusion effects.
  • 1 Introduction: Flydra integrates standard algorithms, hardware, and software into a coherent low-latency multi-target tracking system.Its contribution is the integration of these components with realtime operation rather than reliance on wholly new components.
  • 1 Introduction: The tracker estimates the posterior state of all targets recursively from observations and motion models.An Extended Kalman Filter approximates the estimate, while Nearest Neighbor Standard Filter data association links observations to target models and handles targets entering or leaving the volume.
  • 1 Introduction: Flydra differs from earlier systems through large-volume tracking, explicit data association, general multi-view geometry, and measured latency.These design choices address low-contrast detections, more than three cameras, and comparison with systems lacking latency measurements.

2 2D feature extraction

The 2D feature-extraction stage converts camera images into compact target features using background subtraction. These features are designed for fast realtime processing and can be saved instead of full images for later 3D reconstruction.

  • 2 2D feature extraction: The system extracts feature points from images using an elaborated background-subtraction algorithm.Because targets occupy only a few pixels, each feature point generally characterizes one camera view of a target.
  • 2 2D feature extraction: Each camera contributes image-plane coordinates and local image statistics to the observation set.The feature vector includes u, v, α, β, θ, and ϵ, representing image position and extracted local properties.
  • 2 2D feature extraction: Running Gaussian-average background subtraction identifies candidate pixels and computes feature area, center, orientation, and eccentricity.Thresholding, local intensity processing, image moments, and lens-distortion correction support the feature measurements.
  • 2 2D feature extraction: The background estimate is updated periodically on a per-pixel basis to accommodate slowly varying illumination.The authors report that the simple scheme is sufficient for their purposes and minimal-latency operation.
  • 2 2D feature extraction: Saving extracted feature vectors rather than full-frame images greatly reduces data storage requirements for later 3D reconstruction.Realtime processing can retain the vectors zj while postponing trajectory reconstruction.

(Figure 4 near here.)

The tracking formulation models target motion and observations within an Extended Kalman Filter framework while treating targets as mostly independent. This reduces computational complexity but lowers accuracy during near-contact periods.

  • Multi target tracking: The system limits target interactions to reduce the computational complexity of multi-target state estimation.Process updates are independent, and observation likelihoods are independent except for slight data-association coupling.
  • Multi target tracking: Near-independent target models make tracking and data association separable into smaller, parallelizable problems.The authors state that this architecture could theoretically support tens or hundreds of targets with sufficient processing units.
  • Multi target tracking: The near-independence assumption reduces tracking accuracy when targets approach or nearly contact each other.The paper suggests that such periods could later be analyzed offline with more sophisticated data-association methods.
  • Multi target tracking: An Extended Kalman Filter estimates each target’s state and covariance from prior estimates and incoming observations.The filter is used for nonlinear processes with additive Gaussian noise under the paper’s Gaussian-noise assumption.
  • Multi target tracking: The target state contains 3D position and velocity, while a linear constant-velocity model describes system dynamics.Maneuvering is modeled as process noise, and frequent observations update the estimate to capture more complex trajectories.

(Figure 5 near here.)

The tracking system resolves multi-target data association with an efficient nearest-neighbor strategy and estimates target states using an extended Kalman filter. Its design gates observations, models camera projections and noise, and addresses trajectory merging and missing data through explicit processing rules.

  • State estimation: A nonlinear projective-camera observation model within an extended Kalman filter represents image-plane error as Gaussian noise and estimates each target’s state and error.Triangulation occurs implicitly, while position uncertainty is larger along the camera-ray direction.
  • Failure handling: When all views are occluded or low contrast, the filter uses the prior prediction, causing error covariance to grow by the process covariance without new observations.Filter parameters were selected through educated guesses and iterative trial and error on several observed trajectories.
  • Data association: The system marginalizes observation likelihood across hidden data-association hypotheses, then limits computation with the Nearest Neighbor Standard Filter run independently for each target.Considering all hypotheses across multiple time steps would create a combinatorial explosion.
  • Data association: The NNSF data-association matrix has one column per active target and one row per camera, with entries indicating assigned feature indices or null observations.Assignments are computed from predicted target locations and camera-returned features.
  • Failure handling: A single-pass rule was added to prevent target trajectories from merging when animals share observations, a problem previously encountered during hummingbird territorial fights.The supplied passage introduces the rule but truncates its implementation details.
  • Observation processing: Predicted target positions gate unlikely 2D image observations, reducing the candidate measurements considered by the generative appearance model.This gating step is part of the NNSF implementation and limits computation.

(Figure 6 near here.)

Flydra associates camera-derived feature points with existing Kalman models, false positives, or new targets while managing targets that become invisible or reappear. It uses predicted locations, geometric gating, and reprojection-based initialization to support simultaneous tracking.

  • Target management: The data-association algorithm assigns each incoming feature to an existing Kalman model, a new model, or a false-positive null target.Old models are deleted when targets leave the tracking volume or lack recent observations.
  • Target management: Target visibility changes are handled within the Bayesian update model by initializing new targets or removing existing ones.The motion model includes both target entry and target removal.
  • Target management: Unclaimed detections are triangulated across camera subsets, and the best sufficiently accurate 3D point initializes a Kalman filter with zero velocity.Tracking stops when estimated error exceeds a threshold, commonly after the target leaves the tracking area.
  • Assignment: For each target, the assignment function uses image points from all cameras and prior information to select camera-specific observations.Each assignment-vector component is either null or a column index, and no camera views the same target more than once.
  • Gating: Detections must lie within a threshold Euclidean distance of the predicted projected target location before further likelihood evaluation.The projection is obtained from the expected 3D target position and the camera matrix.
  • Gating: Additional likelihood checks use detected-object area and Mahalanobis distance from the expected 3D position along the camera ray.The ray-based distance is minimized using the predicted state and covariance.

4 Camera and lens calibration

Flydra requires accurate camera calibration because changing camera subsets can otherwise introduce artifacts into reconstructed trajectories. Calibration uses corresponding 2D/3D points and estimates camera geometry, including possible nonlinear distortion parameters.

  • Calibration: Calibration misalignment can introduce artifacts when target visibility shifts between different camera subsets.The system therefore treats good camera calibrations as critical for tracking accuracy.
  • Calibration: Moving an LED point source or freely flying fly through the tracking volume generates corresponding 2D/3D points for an overdetermined calibration solution.The resulting reconstruction is accurate up to scale, rotation, and translation before camera centers determine those transformations.
  • Calibration: Calibration can be iterated to estimate nonlinear camera parameters such as radial distortion.Camera centers from DLT or direct measurements help determine scale, rotation, and translation.

5 Implementation and evaluation

The authors implemented Flydra in three hardware configurations for fruit flies and hummingbirds, using different camera counts, frame rates, arenas, and computer counts. Reconstruction accuracy was assessed through reprojection and physical measurements.

  • Implementations: Three systems used five cameras at 100 fps, eleven cameras at 60 fps, or four cameras at 200 fps for different animal-tracking arenas.The configurations covered fruit flies in 0.3m x 0.3m x 1.5m and 2m-diameter x 0.8m-high arenas, plus hummingbirds in a 1.5m x 1.5m x 3m arena.
  • Implementation: Flydra was implemented in Python with open-source software components and Intel Pentium 4 or Core 2 Duo computers.The system used specialized libraries and networking software for image processing and tracking.
  • Evaluation: Mean reprojection error was less than one pixel, usually below 0.5 pixels for most cameras, while physical triangulation measurements were within 4 percent.Accuracy was verified both by projecting reconstructed 3D estimates back into camera images and by comparison with physical measurements.

(Figure 7 near here.)

The system’s measured 3D reconstruction latency was 39 msec, with image transfer and feature extraction accounting for most of the delay.

  • 39 msec was the median 3D reconstruction timestamp measured from frame-trigger acquisition to computed state vector.The measurement used synchronized computer and microcontroller clocks with a 3 GHz Intel Core 2 Duo central computer.
  • 19.5 msec was the lower bound for image transfer across the IEEE 1394 bus.The authors note that alternative technologies such as Gigabit Ethernet or Camera Link could presumably reduce this component.
  • 26–32 msec was the combined duration of image acquisition and 2D feature extraction.2D feature extraction alone took 6–12 msec in the measured system.

6 Experimental Possibilities

Flydra’s realtime 3D tracking supports experiments that connect an animal’s movement to stimuli, trigger high-speed imaging, reconstruct experienced visual scenes, and extend tracking to naturalistic environments.

  • Online position and velocity estimates can trigger 1024x1024-pixel cameras operating at 6000 frames per second during selected animal movements.The trigger can use current position or recent positional history, including to capture specific maneuvers.
  • Low-latency 3D state estimates enable visual stimuli to be modulated as an animal moves through a virtual-reality environment.Near-IR illumination and high-pass camera filters help prevent moving visual stimuli from affecting tracking.
  • Estimated animal orientation combined with a 3D environment model can reconstruct an approximate view through a fly’s compound eyes.The example assumes the head is tangent to travel direction and the roll angle is zero.
  • The configurable system makes large-scale tracking in naturalistic environments feasible for studying flies and other animals.The paper connects this possibility to understanding the natural history of flies.

7 Effect of contrast on speed regulation in Drosophila

The contrast experiment found that Drosophila flight speed became faster and more variable at low contrast, whereas high contrast supported regulation around a constant speed.

  • Earlier electrophysiological studies found a quadratic relationship between membrane potential and luminance contrast at low contrast levels.These findings concerned mechanisms of motion detection in flies.
  • At Michelson contrast ≥1.6, Drosophila regulated flight speed to a mean of 0.15 m/s with a standard deviation of 0.07.As contrast decreased, both mean speed and speed variability increased, suggesting reduced visual-feedback-based regulation.
  • Low-contrast Drosophila flight was faster and more variable than honey-bee flight below 0.16 Michelson contrast.The paper lists several possible explanations, including differences in arena imperfections, contrast sensitivity, adaptation, or motion detection mechanisms.
  • High-contrast flight speed was regulated around a constant value in both Drosophila and honey bees above 0.16 Michelson contrast for Drosophila.The authors suggest that insects regulate flight speed around a set point using visual information at these contrast values.

Appendix A Extended Kalman filter

The appendix expresses the tracking update using an Extended Kalman filter, with linear process dynamics and a nonlinear observation model handled through a Jacobian.

  • The Extended Kalman filter approximates the Bayesian tracking solution using linear process dynamics because only the observation process is nonlinear.The process dynamics are therefore specified with matrix A.
  • The filter predicts a priori state values from the previous frame’s posterior estimates.This prediction forms the state estimate before incorporating the current observation.
  • The gain term K weights the innovation between the a priori state estimate and the observation.The innovation is the difference used to incorporate new measurements.
  • The observation matrix H_t is the Jacobian of the observation function evaluated at the estimate.It provides the local linearization required for the nonlinear observation update.
  • Posterior estimates are produced after the observation update.

Appendix B Triangulation

Appendix B describes how Flydra reconstructs 3D position and body orientation from two or more calibrated camera views. It combines projective camera geometry with homogeneous linear triangulation solved by singular value decomposition.

  • 2D camera points are converted into 3D locations by finding the best intersection of rays from the corresponding camera centers.The calculation uses a linear least-squares fit over the rays defined by two or more image points.
  • Homogeneous image coordinates are converted to Cartesian coordinates using the function H, with u_i = r_i/t_i and v_i = s_i/t_i.
  • The calibration matrix P_i projects a 3D homogeneous point X into camera i’s image coordinates.Inhomogeneous 3D coordinates are represented as ratios of the homogeneous coordinates.
  • Homogeneous linear triangulation combines equations from multiple cameras and solves for X using singular value decomposition.A similar line-fitting approach reconstructs the longitudinal orientation axis of insects or birds from image coordinates and body-axis angles.
Loading 1001.4297v1…