PaperlayerView exact revision on arXiv

Source-linked AI summary

Event-based Moving Object Detection and Tracking

Anton Mitrokhin, Cornelia Fermuller, Chethan Parameshwara, Yiannis Aloimonos

arXiv:1803.04523v3cs.CV

TL;DR

The paper addresses independent-motion detection and tracking from asynchronous event cameras, whose low resolution, noise, and temporal structure require specialized processing. It introduces a timestamp-based time-image representation and parametric event-cloud motion compensation, then demonstrates tracking and detection in challenging fast-motion and lighting conditions. The reported framework operates without external sensors, feature tracking, or explicit optical-flow computation, while remaining efficient and parallelizable.

  • Problem

    Event cameras provide high temporal resolution, low latency, and light sensitivity, but their low resolution, noise, and asynchronous streams create challenges for motion analysis.

  • Method

    The paper uses a timestamp-based time-image representation and a parametric model to globally compensate camera motion and detect objects through temporal motion inconsistencies.

  • Results

    The framework detects and tracks independently moving objects in challenging fast-motion, low-light, flashing-strobe, and lighting-variation sequences without external sensors, feature tracking, or explicit optical flow.

  • Takeaways & Limitations

    The event-only framework provides a global, parallelizable approach to motion segmentation and tracking that can be ported to GPUs or FPGAs for lower latency.

Abstract

from arXiv · show

Event-based vision sensors, such as the Dynamic Vision Sensor (DVS), are ideally suited for real-time motion analysis. The unique properties encompassed in the readings of such sensors provide high temporal resolution, superior sensitivity to light and low latency. These properties provide the grounds to estimate motion extremely reliably in the most sophisticated scenarios but they come at a price - modern event-based vision sensors have extremely low resolution and produce a lot of noise. Moreover, the asynchronous nature of the event stream calls for novel algorithms. This paper presents a new, efficient approach to object tracking with asynchronous cameras. We present a novel event stream representation which enables us to utilize information about the dynamic (temporal) component of the event stream, and not only the spatial component, at every moment of time. This is done by approximating the 3D geometry of the event stream with a parametric model; as a result, the algorithm is capable of producing the motion-compensated event stream (effectively approximating egomotion), and without using any form of external sensors in extremely low-light and noisy conditions without any form of feature tracking or explicit optical flow computation. We demonstrate our framework on the task of independent motion detection and tracking, where we use the temporal model inconsistencies to locate differently moving objects in challenging situations of very fast motion.

I. INTRODUCTION AND PHILOSOPHY

The paper targets independent moving-object segmentation and tracking from a moving event camera, using temporal event information to handle fast motion and difficult lighting. It contributes a time-image representation, event-only motion compensation, an evaluation dataset, and an efficient implementation.

  • Motivation: Event-based sensing offers high temporal resolution, low latency, and low-bandwidth motion signals, but conventional vision struggles with fast motion, low light, and changing illumination.These properties motivate event-based frameworks for robotic visual navigation and real-time motion analysis.
  • Problem: The paper focuses on segmenting and tracking multiple independently moving objects from a moving event camera.It estimates camera ego-motion globally from the event stream and detects objects through inconsistencies in the motion field.
  • Contributions: The paper introduces an event-only, feature-less motion-compensation pipeline for estimating camera motion without relying on image frames.The framework is highly parallelizable and can be ported to GPUs or FPGAs for lower-latency operation.
  • Approach: A new event timestamp-based time-image representation uses the temporal component of events to improve motion-compensation robustness.The representation supports motion-compensated event images in which independently moving objects remain visible for subsequent detection and tracking.
  • Contributions: The authors provide the Extreme Event Dataset with multiple moving objects under low lighting and extreme light variation, alongside quantitative and qualitative evaluation.They also report an efficient C++ implementation intended for open-source release.

II. RELATED WORK

The related work spans event-based clustering, tracking, feature tracking, optical flow, visual odometry, SLAM, and independent-motion detection. The paper positions its approach as global motion compensation using a four-parameter event-cloud model, inspired by point-cloud processing.

  • Event-based tracking: Prior event-based tracking methods follow clusters, Gaussian-mixture models, FPGA noise-removal pipelines, or event-assignment strategies.These approaches generally organize events into tracked regions or clusters.
  • Motion estimation: Other work develops event-based feature trackers, optical-flow estimators, visual odometry, and SLAM under simplified motion, map, pose, depth, or sensor-fusion assumptions.An unrestricted 3D-motion event-based solution is also identified in the prior work.
  • Closest related work: Closely related studies estimate global 3D rotation, use linear iterative segmentation, or detect independent motion from learned static-scene flow and tracked corners.The paper distinguishes its global event-cloud formulation from these related strategies.
  • Positioning: The proposed algorithm fits a four-parameter motion model to an event cloud, compensates camera motion globally, and labels nonconforming clusters as independently moving regions.The parameters represent image-plane shifts, expansion, and 2D rotation; models are also fit to detected regions.
  • Positioning: The method is inspired by Kinect Fusion-style warp-field minimization on point clouds, with detailed algorithmic stages devoted to motion compensation and object detection or tracking.The paper separately describes notation, error-function intuition, motion compensation, and object detection and tracking.

A. Notation

Events are represented as a 3D space-time cloud and warped to compensate motion before projection into event-based image maps. The event-count image measures projected occupancy, while timestamps provide temporal structure and the event polarity is unused.

  • Event representation: An event in a temporal segment is represented by spatial coordinates x, y and timestamp t, with processing performed over a small interval δt.The input cloud is defined in R3, while t0 can be chosen arbitrarily because DVS data is continuous.
  • Warping and projection: The warp field maps events from time t to locations at reference time t0, producing motion-compensated events whose projected density is maximized.Temporal projection reduces the warped event cloud from R3 to R2 for minimization.
  • Event-count image: The event-count image I records how many warped events project into each discretized pixel, and motion compensation can minimize the occupied pixel area.The event density is the number of warped events divided by the number of occupied pixels.
  • Event representation: The event trajectory ξij collects warped events projected onto pixel (i, j), whose cardinality contributes to the event-count image and density calculations.Each trajectory is defined after applying the warp field to the event cloud.
  • Input data: The DVS polarity component is not used because noise at object boundaries makes the binary intensity-change value unreliable.The sensor data is four-dimensional, but this representation uses the spatial and temporal components.

C. Time-image T

The event-count image can confuse events from different edges when projection overlaps, especially during fast motion in textured scenes. The time-image T addresses this by retaining average event timestamps and supplying motion-error gradients.

  • Motivation: Projection overlap in the event-count image can combine events from different edges, a common problem during fast motion in highly textured scenes.The overlap may involve different parts of one object or different objects.
  • Representation: The time-image T stores the average timestamp of events mapped to each discretized pixel by the warp field.It uses event timestamps rather than only projected event occupancy.
  • Representation: Averaging timestamps uses all available DVS events, whereas using only the latest timestamps can reduce performance in low-light conditions.Lower average illumination increases DVS noise because the signal-to-noise ratio depends on illumination.
  • Motion detection: Deviation of a time-image pixel from the mean correlates with the probability that local motion was not compensated.This temporal inconsistency is later used for motion detection.
  • Error metric: The spatial gradient of T provides a global motion-compensation error, decomposed into parameter-specific errors for shift, expansion, and rotation.The gradient-based formulation accounts for local motion inconsistencies in the warped event cloud.

D. Minimization Constraints

Global motion is modeled by a four-parameter warp that transforms event coordinates while preserving timestamps and assuming linear event trajectories within each time slice. The time-image can reveal independently moving objects even when they occupy much of the frame.

  • Motion model: The global warp field uses four parameters to describe camera-induced event-cloud distortion: x-shift, y-shift, expansion, and 2D rotation.The parameters are denoted MG = {hx, hy, hz, θ}.
  • Coordinate transformation: The coordinate transformation changes event positions from {x, y, t} to {x′, y′, t}, leaving timestamps unchanged.The model assumes linear event trajectories within each temporal slice.
  • Motion model: The model parameters represent image-plane shifts, radial expansion toward the image plane, and rotation around the Z axis.Expansion is represented by motion along the depth direction, while rotation produces a circular component.
  • Example: In the example time-image, blue and green encode average timestamps from t0 to t0 + δt, while a separately moving drone occupies a large frame area yet camera compensation succeeds.The example illustrates temporal separation of object motion from camera motion.

V. CAMERA MOTION COMPENSATION

Camera motion compensation first estimates global background motion, then refines it after excluding detected objects and fits motion models to segmented objects for tracking. Optimization uses coarse timestamp-based minimization followed by event-density refinement.

  • Pipeline: The pipeline estimates global background motion with a four-parameter model, detects objects, and refines the background model using only background-region data.Detected objects are excluded from the refinement step before object-specific tracking models are fitted.
  • Constraint: The time-image error gradients can become very small and unreliable when the optimizer is close to the minima.This limitation motivates the subsequent event-count-image refinement stage.
  • Gradient reliability: T provides reliable motion-model gradients even with noise and fast motion when events from different edges overlap during projection.This supports using the time-image for the coarse motion stage.
  • Optimization: The global minimization proceeds in two stages: coarse motion minimization on T followed by fine motion refinement on I.The two stages use timestamp and event-rate information respectively.
  • Coarse compensation: Algorithm 1 performs coarse event-cloud compensation by warping the cloud, generating T, computing image gradients, and updating model parameters by gradient descent.The discretization parameter d is chosen as 0.3 of the DVS pixel size.

B. Fine Global Motion Refinement on I

The method refines global motion compensation by maximizing event-count image density after an initial model-based estimate. It uses parameter perturbations to obtain derivatives and iteratively minimizes the resulting objective.

  • B. Fine Global Motion Refinement on I: Fine motion refinement maximizes the event-count image density D to improve global motion compensation.Because D does not explicitly provide gradients, the method estimates derivatives by varying model parameters.
  • B. Fine Global Motion Refinement on I: Algorithm 2 performs global motion compensation in event space using the event-count image.
  • B. Fine Global Motion Refinement on I: The detection stage observes inconsistencies in the time-image representation T to identify independently moving objects.Detected objects are subsequently tracked with a traditional Kalman Filter.

A. Detection

Detection identifies independently moving objects from pixelwise misalignment with the background, then groups foreground pixels and tracks objects through missed or incorrect detections.

  • A. Detection: Each pixel receives a score ρ(x_i, y_j) in [−1, 1] that measures misalignment relative to the background.
  • A. Detection: Pixels with ρ(i, j) ≤ 0 are classified as background, while pixels with ρ(i, j) > λ form independently moving objects.λ is a predefined minimum confidence value for object classification.
  • A. Detection: Foreground pixels are grouped into objects using simple morphological operations.
  • A. Detection: A real-time detection algorithm uses a constant-acceleration Kalman Filter to handle missing and incorrect detections, especially during occlusion.The detection processing time is less than δt.
  • A. Detection: Object measurements include mean coordinates and motion-compensated model parameters, together with average velocity estimates.

VII. DATASETS AND EVALUATION

The evaluation uses the Extreme Event Dataset, collected with a DAVIS240B sensor in hand-held and quadrotor setups across varied motion, texture, occlusion, and lighting conditions. Experiments emphasize real-time operation and robustness to strobe-induced noise, while GPU acceleration substantially reduces iteration time.

  • VII. DATASETS AND EVALUATION: The Extreme Event Dataset contains over 30 recordings from quadrotor and hand-held setups spanning object sizes, speeds, lighting conditions, and non-rigid camera motions.The dataset includes sequences with abrupt changes in object speed.
  • VII. DATASETS AND EVALUATION: The high-texture, fast-motion experiment shows overlapping tilted event planes and uses the time-image to obtain the correct motion-compensation minimization gradient.
  • VII. DATASETS AND EVALUATION: The quadrotor platform combines a mounted DAVIS240B camera with a customized Qualcomm Flight platform and onboard computer.The fully loaded platform weighs approximately 500 g and uses a four-core Snapdragon APQ8074 ARM CPU reaching 2.3 GHz.
  • VII. DATASETS AND EVALUATION: 10 ms and 7 ms are the average single-iteration times for Algorithms 1 and 2 on one Intel Core i7 3.2 GHz thread.The warp-and-project operation is highly parallelizable for GPU or FPGA implementation.
  • VII. DATASETS AND EVALUATION: 0.01 ms and 0.003 ms are the average GPU iteration times for Algorithms 1 and 2, corresponding to 1000X and 2333X speed-ups.The minimization converges in fewer than 30 iterations on average, yielding faster-than-real-time computation.
  • VII. DATASETS AND EVALUATION: The dataset also includes fast-drone, multiple-object, background-through-net, and occluded sequences to test varied tracking conditions.
  • VII. DATASETS AND EVALUATION: The ’Lighting variation’ sequence uses a 1–2 Hz strobe in a dark room to generate substantial event-sensor noise.It is described as extremely challenging and otherwise similar to the fast-moving-drone sequence.

C. Metrics and Evaluation

The evaluation measures detection success against hand-labeled RGB-frame ground truth and examines performance across challenging motion, occlusion, and noise conditions. Failures arise when independently moving objects are indistinguishable from the background or when severe noise overwhelms the time image.

  • Metrics: Success rate averages, across sequences, the percentage of detected objects overlapping their RGB-frame ground-truth regions by at least 50%.Ground truth was obtained by hand-labeling RGB frames, with a separate score computed for each corresponding time slice.
  • Challenging conditions: High performance was achieved on “What is a Background?” despite partial occlusion, because the textured net supported robust camera-motion compensation.
  • Challenging conditions: Crossing object paths challenged tracking, although the Kalman filter often distinguished objects using differences in their previous motion states.
  • Failure cases: Detection failed when a fast-moving drone had nearly zero motion relative to the background, even though camera motion was successfully compensated.
  • Failure cases: Severe noise made both motion compensation and detection unreliable when too little background remained visible in the time image.The “Lighting variation” sequence illustrates this failure mode.

VIII. CONCLUSIONS

The paper presents event-based-only motion segmentation and tracking for unconstrained 3D motion and unknown scenes. Experiments under fast motion, multiple moving objects, and lighting variation demonstrate the method’s usefulness, while future work targets more complete 3D motion estimation and clustering.

  • Conclusions: The paper presents an event-based-only method for motion segmentation under unconstrained 3D motion and unknown scenes.
  • Conclusions: Experiments demonstrate usefulness under fast motion, multiple moving objects, and lighting variations.
  • Future work: Future work will extend clustering and segmentation toward iterative 3D motion estimation while detecting objects moving similarly to the camera.
  • Evaluation data: The EED dataset includes recordings with one, two, and three moving objects, strobe lighting, low lighting, occlusion, and ambiguous backgrounds.
  • Evaluation data: The evaluation pipeline is reported on the proposed dataset in Table I.
Loading 1803.04523v3…