Source-linked AI summary
Asynchronous, Photometric Feature Tracking using Events and Frames
Daniel Gehrig, Henri Rebecq, Guillermo Gallego, Davide Scaramuzza
TL;DR
Event-camera feature tracking lacks stable correspondences because the same scene pattern produces motion-dependent events, whereas frames provide motion-independent intensity information. The paper extracts features on frames and tracks them asynchronously with events using a maximum-likelihood generative event model, achieving subpixel-accurate, longer tracks than the state of the art. The method is limited by event-model noise and assumptions such as locally uniform optical flow.
Problem
Motion-dependent event appearance makes establishing feature correspondences across time difficult, limiting event-based feature tracking.
Method
The tracker extracts features from frames and tracks them asynchronously using events, relating raw frame intensities to events through a maximum-likelihood generative model.
Results
The method produces subpixel-accurate feature tracks that are more accurate and longer than state-of-the-art methods across varied scenes and sequences.
Takeaways & Limitations
Combining frame photometric information with asynchronous events provides low-latency feature tracks across a variety of scenes and motions.
Takeaways & Limitations
The method does not account for severe event noise and dynamic effects and assumes uniform optical flow near features, which breaks down at occlusions and large flow distortions.
Abstract
from arXiv · showhide
We present a method that leverages the complementarity of event cameras and standard cameras to track visual features with low-latency. Event cameras are novel sensors that output pixel-level brightness changes, called "events". They offer significant advantages over standard cameras, namely a very high dynamic range, no motion blur, and a latency in the order of microseconds. However, because the same scene pattern can produce different events depending on the motion direction, establishing event correspondences across time is challenging. By contrast, standard cameras provide intensity measurements (frames) that do not depend on motion direction. Our method extracts features on frames and subsequently tracks them asynchronously using events, thereby exploiting the best of both types of data: the frames provide a photometric representation that does not depend on motion direction and the events provide low-latency updates. In contrast to previous works, which are based on heuristics, this is the first principled method that uses raw intensity measurements directly, based on a generative event model within a maximum-likelihood framework. As a result, our method produces feature tracks that are both more accurate (subpixel accuracy) and longer than the state of the art, across a wide variety of scenes.
1 Introduction
Event and standard cameras provide complementary measurements for low-latency feature tracking: events respond asynchronously to brightness changes, while frames provide direct intensity measurements. The paper combines them through a maximum-likelihood generative event model to produce more accurate and longer tracks.
- 1 Introduction: Event cameras respond asynchronously to brightness changes with 1-microsecond latency but do not provide absolute intensity measurements.Standard cameras provide direct intensity measurements, creating complementary sensing capabilities.
- 1 Introduction: A maximum-likelihood generative event model uses frame photometric information to address data association between events and pixels.The approach is designed to combine both intensity measurements to maximize tracking accuracy and track age.
- 1 Introduction: Frame-based trackers cannot operate in the blind time between consecutive frames and process all pixels even without scene motion.Event cameras acquire only relevant information asynchronously, filling this blind time.
- 1 Introduction: The proposed tracker extracts corners from frames and subsequently tracks them using only events for high-temporal-resolution feature tracks.This uses the asynchronous, high-dynamic-range, and low-latency properties of events.
- 1 Introduction: Compared with state-of-the-art methods, the tracker produces more accurate and longer feature tracks across man-made and natural scenes.The evaluation uses scenes from the publicly available Event Camera Dataset.
2 Related Work
Event-camera tracking is motivated by asynchronous, high-dynamic-range sensing but is complicated by motion-dependent feature appearance. The paper uses raw frame intensity gradients rather than binary edge templates or explicit event-template correspondences.
- 2 Related Work: Event cameras and standard cameras differ in output: the former emit asynchronous brightness-change events, while the latter provide fixed-rate frames.A DAVIS combines both outputs, including events occurring between frames.
- 2 Related Work: Prior event-camera methods include event-based extensions of Harris and FAST detectors, optical-flow-distribution detectors, and binary feature-template trackers.These methods target challenging low-power, high-speed, and high-dynamic-range scenarios.
- 2 Related Work: The method most closely related to this work detects Canny edge patches around frame-based Harris corners and aligns them with events using ICP.That approach treats the Canny edge map as a proxy for the grayscale pattern under a strong-edge assumption.
- 2 Related Work: Unlike prior point-set-template approaches, this method uses the spatial gradient of the raw intensity image and implicitly establishes event correspondences through image registration.It also accounts for the strength of the edge pattern causing events.
3 The Challenge of Data Association for Feature Tracking
Event-camera feature appearance changes with motion direction because events encode temporal intensity changes, making correspondences across time difficult. Absolute intensity information can provide a time-invariant feature map for indirect association, while the method models motion-dependent brightness increments.
- 3 The Challenge of Data Association for Feature Tracking: Event-camera feature appearance varies with motion because the sensor responds to temporal intensity changes from moving image edges.This changing appearance makes data association between events at different times difficult.
- 3 The Challenge of Data Association for Feature Tracking: Absolute intensity measurements provide a time-invariant feature map that can support indirect event correspondences.Using this map additionally requires continuously estimating the pattern’s optic flow.
- 3 The Challenge of Data Association for Feature Tracking: The generative model predicts brightness increments from a frame and optic flow for comparison with event-derived increments.The figure represents unchanged pixels in gray and increases or decreases in bright and dark, respectively.
4 Methodology
The method registers event-derived brightness increments against frame-derived predictions, estimating warp and optic flow asynchronously through maximum-likelihood photometric alignment. It initializes frame features, accumulates event patches, optimizes their residual, and updates tracks independently as events arrive.
- Brightness-Increment Images from Events and Frames: Brightness increments from moving edges are modeled from the frame gradient and optical flow, vanishing when motion is parallel to the edge.The model uses the dot product between the spatial gradient and velocity over displacement v∆τ.
- Optimization Framework: The tracker estimates registration parameters p and velocity v by minimizing the photometric difference between observed event increments and warped frame predictions.Frames provide feature templates, while events supply later brightness increments for image registration over small patches with constant patch flow.
- Optimization Framework: A maximum-likelihood Gaussian residual model converts likelihood maximization into minimizing an L2 photometric residual, followed by unit-norm patch comparison to cancel C and ∆τ.The normalized objective depends only on the direction of feature velocity, while rigid-body image-plane warps are optimized with nonlinear least squares.
- Implementation: Algorithm 1 detects Harris corners, initializes intensity-gradient patches, accumulates incoming event polarities, optimizes after Ne events, and resets each patch for continued tracking.Independent patches are tracked asynchronously; monitoring minimum cost values provides a criterion for detecting track loss and initializing new tracks.
- Discussion of the Approach: The frame generates a velocity-dependent gradient template, allowing the method to estimate both feature warping and optic flow while using full gradient information instead of binary Canny edges.This motion-dependent template is designed to fit event data and supports tracking less salient patterns.
5 Experiments
The experiments evaluate the tracker on simulated, real, and HDR-reconstructed data, measuring accuracy, feature survival, and robustness against prior methods. Across these settings, the method achieves subpixel accuracy, longer tracks, and improved behavior in challenging scenes, while real-data error remains higher because of event-modeling errors.
- Experimental setup: The evaluation covers simulated scenes with controlled ground truth, real scenes containing occlusions, parallax, and illumination changes, and frames reconstructed from events for HDR tracking.Synthetic data assesses accuracy limits; real data tests challenging visual effects; reconstructed frames test high-dynamic-range tracking.
- Simulated data: 0.4 pixel average error is achieved on simulated data under noise-free conditions, with residual error likely caused by linearization in the motion model.Feature tracks are discarded when they leave the field of view or when registration error exceeds 1.6.
- Real data: Our method outperforms and in both tracking accuracy and track length across simple, highly textured, and natural scenes.In simple black-and-white scenes, it is on average twice as accurate and produces tracks almost three times longer than.
- Real data: The method remains accurate in highly textured scenes and is more accurate than on natural scenes, where exhibits drift from changing event appearance.The proposed tracker uses a time-invariant template and generative model, while also exploiting full brightness-gradient information.
- Limitations: Real-data tracking error is larger than synthetic-data error, likely because the event model does not fully capture noise and dynamic effects such as unequal polarity thresholds.Despite this limitation, the tracker maintains subpixel accuracy and consistently produces more accurate and longer tracks than previous methods.
- Objective function comparison: The proposed cost function is convex in tested simple and highly textured scenes, whereas has local minima and broad attraction basins that make registration difficult.The resulting tracker follows KLT ground truth accurately, while shows jumping behavior and early divergence.
- HDR tracking: Event-reconstructed HDR intensity images provide tracking information in overexposed regions, and only a limited number of reconstructions are needed because features can be tracked for several seconds.Standard cameras provide 60 dB dynamic range, compared with 140 dB for event cameras.
6 Discussion
The method can track features from event-reconstructed intensity images, including high-dynamic-range scenes. Its main limitations are approximate event prediction, uniform-flow assumptions, and reliance on Euclidean warps.
- The event-prediction model omits severe dynamic effects and noise.
- Uniform optical flow near features fails at occlusions and under large flow distortions, including motion along the camera’s optical axis.
- The demonstrated Euclidean warp was chosen because it was more stable than more complex models such as affine warps.
7 Conclusion
The paper combines frame-based feature extraction with asynchronous event-based tracking through a generative event model. Across varied sequences, it reports more accurate and longer tracks than the state of the art.
- The tracker extracts features from frames and subsequently tracks them asynchronously using events.
- A generative event model relates events directly to frame pixel intensities.
- The method produces more accurate, subpixel-accurate, and longer feature tracks than the state of the art across varied sequences.
8 Supplementary Material
The supplementary experiments examine event accumulation, patch size, dataset-dependent tracking, and robustness under low-light and abrupt illumination changes. They support adaptive event integration and a 25×25-pixel patch compromise between accuracy and feature age.
- 8.2 Sensitivity with respect to the Number of Events in a Patch: The optimal event count depends on scene texture and motion, including motion parallel to edges that suppresses some events.
- 8.2 Sensitivity with respect to the Number of Events in a Patch: Event accumulation affects the objective-function shape and its minimizer: too few events lack appearance information, while too many wash it out.
- 8.2 Sensitivity with respect to the Number of Events in a Patch: The tracker uses the estimated unit velocity to compute the event count processed at each time step.
- 8.3 Sensitivity with respect to Patch Size: Larger patches improve accuracy, whereas medium-to-smaller patches produce longer tracks; 25×25 pixels balances accuracy and feature age.
- 8.4 Feature Tracking on All Datasets: Black-and-white scenes yield the longest tracks, reaching 25 s for the checkerboard dataset, while high-contrast and high-texture scenes generally outlast natural scenes.
- 8.5 Feature Tracking in Low-Light and with Abrupt Light Changes: During light-off intervals, the tracker maintains stable features and resumes after illumination returns, whereas KLT loses track immediately.