PaperlayerView exact revision on arXiv

Source-linked AI summary

Speed Invariant Time Surface for Learning to Detect Corner Points with Event-Based Cameras

Jacques Manderscheid, Amos Sironi, Nicolas Bourdis, Davide Migliore, Vincent Lepetit

arXiv:1903.11332v2cs.CV

TL;DR

Event-based corner detection is challenging because event data differ from intensity images, making simple adaptations of frame-based detectors perform poorly. The paper uses a speed-invariant Time Surface and Random Forest to recognize corner-generated events; the authors report greater robustness to abrupt direction changes and strong tracking results with simple nearest-neighbor matching.

  • Problem

    Simple extensions of frame-based corner detectors perform poorly on event data, whose patterns vary with corner motion and whose event-level labels are difficult to build manually.

    Source: §1 · p. 2, §4.2 · p. 4, §4.4 · p. 6

  • Method

    The method classifies individual events with a Random Forest using a speed-invariant Time Surface representation.

    Source: §1 · p. 2, §4.3 · p. 5

  • Results

    The authors report more robust corner detection under abrupt direction changes and state-of-the-art results when combining the detector with simple nearest-neighbor tracking.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The method produces stable, repeatable corners and achieves good accuracy with a simple tracking algorithm.

    Source: §7 · p. 8

    Evaluation against Harris corners from gray-level images can penalize event-based features that do not match those corners, while favoring this method because it was trained from Harris corners.

    Source: §5 · p. 6

Abstract

from arXiv · show

We propose a learning approach to corner detection for event-based cameras that is stable even under fast and abrupt motions. Event-based cameras offer high temporal resolution, power efficiency, and high dynamic range. However, the properties of event-based data are very different compared to standard intensity images, and simple extensions of corner detection methods designed for these images do not perform well on event-based data. We first introduce an efficient way to compute a time surface that is invariant to the speed of the objects. We then show that we can train a Random Forest to recognize events generated by a moving corner from our time surface. Random Forests are also extremely efficient, and therefore a good choice to deal with the high capture frequency of event-based cameras ---our implementation processes up to 1.6Mev/s on a single CPU. Thanks to our time surface formulation and this learning approach, our method is significantly more robust to abrupt changes of direction of the corners compared to previous ones. Our method also naturally assigns a confidence score for the corners, which can be useful for postprocessing. Moreover, we introduce a high-resolution dataset suitable for quantitative evaluation and comparison of corner detection methods for event-based cameras. We call our approach SILC, for Speed Invariant Learned Corners, and compare it to the state-of-the-art with extensive experiments, showing better performance.

1. Introduction

Event cameras support fast, low-power vision, but detecting feature points from their asynchronous events remains challenging. SILC learns to classify events using a speed-invariant Time Surface, and its detector works with simple tracking for state-of-the-art results.

1. Introduction

  • Adapted frame-based detectors can be unstable over time because they do not account for event-data noise patterns and responses to direction changes.Source: §1 · pp. 1–2
  • The method combines a speed-invariant Time Surface with a learned event classifier to detect corners more robustly across motion changes and noise.The classifier labels individual events as generated by a moving feature point or not.Source: §1 · p. 2
  • The Random Forest implementation processes up to 1.6 · 10^6 events per second on a single CPU.Random Forests are chosen for efficiency without a GPU, which the authors consider counterproductive for low-power applications and sparse asynchronous events.Source: §1 · p. 2
  • The authors release a high-resolution benchmark dataset for evaluating event-based corner detectors.Source: §1 · p. 2
  • A simple nearest-neighbor tracker combined with the detector achieves state-of-the-art results, supported by the temporal continuity of events.Source: §1 · p. 2

2. Event-based cameras

Event-based cameras record asynchronous pixel-level changes, providing microsecond-scale temporal resolution, high dynamic range, and suitability for low-power applications. SILC can run without gray-level information, and the paper releases a dataset captured at HVGA resolution.

2. Event-based cameras

  • Each pixel emits an event when its log-luminosity change crosses a threshold; events carry location, timestamp, and polarity.Source: §2 · p. 2
  • SILC requires no gray-level information at runtime, although intensity information was used to build its training dataset.Source: §2 · p. 2
  • The released dataset is the first event-based dataset acquired with an HVGA (480×360) sensor, and the authors report that their algorithm suits real-time, high-data-rate applications.Source: §2 · p. 2

3. Related work

Prior event-based feature detectors either match local event patterns or adapt frame-based corner tests to event representations. Related work also explores event-learning pipelines that accumulate events into dense inputs or process them event by event.

3. Related work

  • Template-matching detectors can be efficient, but are sensitive to noise and require careful parameter tuning.Examples include Gaussian-kernel blob detection and corners identified from intersections of local velocity planes.Source: §3 · pp. 2–3
  • Frame-based detector adaptations build local event representations before applying classical corner tests, but can lose accuracy when feature motion direction changes.Examples include Harris on accumulated events and FAST on Time Surfaces; event-corner appearance depends on motion, unlike the assumed constant appearance in frame-based vision.Source: §3 · p. 3
  • Gray-level-assisted methods detect features in images and track them with events, but retain frame-based limitations and require gray-level-capable sensors or sensor synchronization.By contrast, the paper says its method needs only a purely event-based sensor at runtime.Source: §3 · p. 3
  • Many event-learning methods accumulate events into dense representations for standard vision models, adding latency through an artificial accumulation period.Examples include histograms for steering-angle or optical-flow prediction and dense descriptors for recognition.Source: §3 · p. 3
  • Event-by-event approaches preserve input timing, while spiking neural networks can be difficult to scale to high event rates and high-resolution inputs.The paper applies its classifier event-by-event while running on a standard CPU.Source: §3 · p. 3
  • Frame-based keypoint learning includes decision-tree and ensemble methods as well as deep models for detecting stable points or jointly predicting points and descriptors.Source: §3 · p. 4

4. Method

The method represents events with a local time surface, normalizes it to reduce speed-dependent variation, and uses it to train a classifier for corner events. It also builds labeled training data from graylevel corner detections.

4.2. Speed Invariant Time Surface

  • The standard Time Surface varies with corner speed, direction, and contrast; on the Speed Invariant Time Surface, all edges have the same slope, and it is identical for two speeds.In the described experiment, the slope has a total length of 5 pixels, equal to parameter r; after the slope the surface is constant at r + 1, and its values remain between 0 and 2r + 1.Source: §4.2 · pp. 4–5, Figure 3 · p. 5
  • The Speed Invariant Time Surface keeps one value per pixel and polarity, decrementing neighboring values so its wave profile is independent of contour speed.For each event, values in a neighborhood are decremented when they exceed the event location's value, which is then set to (2r + 1)^2.Source: §4.2 · p. 4

4.3. Learning to Detect Corner Points from Events

  • A Random Forest classifies each event from an n×n patch of the updated surface, predicting whether it is a feature event.The forest is selected for efficiency; its trees use decision stumps, and leaf outputs give the probability that a sample is a feature point.Source: §4.3 · p. 5

4.4. Building a Dataset for Event-based Corner Detection

  • Because manual labeling of millions of events per second is impractical, the dataset uses graylevel measurements to assign positive and negative event labels.The method applies Harris at each event’s location; recording well-contrasted geometric patterns helps reduce outliers, and the authors report generalization to more complex scenes.Source: §4.4 · p. 6
  • The training set labels events using graylevel Harris detections, with geometric patterns chosen to reduce noise-related outliers.For DAVIS, corners are detected in regularly sampled frames; events within 2 pixels of a detected corner during a 5 ms interval are labeled as corner events.Source: §4.4 · p. 6

5. Evaluation of Event-based Detectors

The evaluation uses established event-camera recordings and a new higher-resolution dataset, assessing detected features through tracking and reprojection accuracy rather than relying only on gray-level keypoints.

Datasets

  • The HVGA ATIS Corner Dataset targets feature detection and tracking with seven increasingly difficult planar-pattern sequences, complementing the more generic Event-Camera dataset.The Event-Camera dataset uses DAVIS recordings and evaluates boxes, shapes, hdr, and dynamic subsets; the simplest shapes subset trains the model.Source: §5 · p. 6

Evaluation metrics

  • Gray-level Harris matching can penalize stable event-only features, while a track-localization metric does not assess whether the feature follows its correct trajectory.Source: §5 · p. 6

Evaluation protocol

  • The evaluation combines detectors with nearest-neighbor tracking, then measures track accuracy using reprojection error.Source: §5 · p. 7
  • For planar scenes, reprojection error comes from RANSAC-estimated homographies between feature correspondences; for 3D scenes, it comes from triangulated tracks and available poses.The 3D protocol also reports the percentage of tracks with error below 5 pixels, and the methods are compared by computational time.Source: §5 · p. 7

6. Experiments

Experiments compare SILC with event-based detectors and standard Time Surfaces using tracking, reprojection error, and runtime. SILC tracks corners longer with low reprojection error, generalizes across sensors, and is reported to run in real time.

Experimental setup

  • Tests compare SILC with publicly available evHarris, evFast, and Arc implementations, applying a trail filter before each detector.SILC uses radius r = 6, patch size n = 8, and a 10-tree Random Forest.Source: §6 · p. 7

Ablation study

  • The Speed Invariant Time Surface Random Forest has lower reprojection error than the standard Time Surface Random Forest, which generalizes poorly and produces noisy detections outside the simple chessboard.Source: §6 · p. 7, Table 1 · p. 7

HVGA ATIS Corner Dataset

  • SILC tracks corners longer with low reprojection error and resists noise in complex scenes; evFast has slightly lower error but shorter tracks.evHarris and Arc produce noise-sensitive, edge-driven tracks described as poorly reliable.Source: §6 · p. 7, Figure 7 · p. 8, Figure 6 · p. 8

Event-Camera Dataset

  • On the lower-resolution, noisier Event-Camera dataset, SILC has the lowest reprojection error and generalizes across sensors; retrained SILC with nearest-neighbor tracking performs comparably to complex trackers.Source: Table 2 · p. 8, §6 · p. 8

Computational time

  • The computational-time comparison reports that SILC runs in real time despite the HVGA sensor's high data rate.Source: Table 3 · p. 8

7. Conclusion and Future Work

The conclusion presents SILC as an efficient, accurate learning approach for event-based corner detection, with stable and repeatable corners and a speed-invariant Time Surface.

Conclusion

  • SILC produces more stable and repeatable corners than prior methods and achieves good accuracy when paired with simple tracking.Source: §7 · p. 8
  • Its Time Surface representation is invariant to the object's local speed.Source: §7 · p. 8

Future work

  • Future work proposes applying the Speed Invariant Time Surface to low-latency object detection and relocalization.Source: §7 · p. 8

Appendix

The appendix table describes the cameras used in the paper and identifies the Gen3 ATIS as the ground-truth camera.

Camera descriptions

  • The camera-description table identifies the Gen3 ATIS as the camera used to generate ground truth.Source: Table 4 · p. 11

Related papers

Loading 1903.11332v2…