PaperlayerView exact revision on arXiv

Source-linked AI summary

Focus Is All You Need: Loss Functions For Event-based Vision

Guillermo Gallego, Mathias Gehrig, Davide Scaramuzza

arXiv:1904.07235v1cs.CVcs.LGcs.RO

TL;DR

Event-based vision needs objective functions that reliably measure alignment of asynchronous events during motion compensation. This paper introduces and categorizes twenty two focus loss functions, relates them to shape-from-focus, and evaluates them across event-based tasks. Variance, gradient, and Laplacian-based losses are among the most accurate, while runtime depends strongly on implementation.

  • Problem

    Motion compensation requires suitable objective functions for measuring event alignment while using the asynchronous timestamps and polarity of event data.

  • Method

    The paper extends motion compensation with twenty two image-based focus loss functions, organizes them by edge strength and dispersion, and connects them to shape-from-focus and autofocus metrics.

  • Results

    Variance, derivative-based, and high-pass losses achieve less than 2.6% error, while variance is fastest at 17 µs; the functions are also applied to depth and optical-flow estimation.

  • Takeaways & Limitations

    Focus loss functions provide a framework for applying mature image-processing analysis tools to event alignment and for estimating motion-related quantities from events.

  • Takeaways & Limitations

    Reported runtimes are illustrative because implementation plays a major role and the code was not optimized or accelerated with function approximation or GPU warping.

Abstract

from arXiv · show

Event cameras are novel vision sensors that output pixel-level brightness changes ("events") instead of traditional video frames. These asynchronous sensors offer several advantages over traditional cameras, such as, high temporal resolution, very high dynamic range, and no motion blur. To unlock the potential of such sensors, motion compensation methods have been recently proposed. We present a collection and taxonomy of twenty two objective functions to analyze event alignment in motion compensation approaches (Fig. 1). We call them Focus Loss Functions since they have strong connections with functions used in traditional shape-from-focus applications. The proposed loss functions allow bringing mature computer vision tools to the realm of event cameras. We compare the accuracy and runtime performance of all loss functions on a publicly available dataset, and conclude that the variance, the gradient and the Laplacian magnitudes are among the best loss functions. The applicability of the loss functions is shown on multiple tasks: rotational motion, depth and optical flow estimation. The proposed focus loss functions allow to unlock the outstanding properties of event cameras.

1. Introduction

Event cameras capture asynchronous brightness changes with high dynamic range, microsecond-scale temporal resolution, and no motion blur. The paper extends motion compensation with twenty two focus loss functions, connects them to shape-from-focus, and evaluates their accuracy and computational effort for event-based vision.

  • Event cameras asynchronously encode pixel brightness changes as events, offering 140 dB dynamic range, microsecond-scale temporal resolution, and no motion blur.These properties target challenging high-speed and high-dynamic-range scenarios.
  • Motion compensation searches for image-plane point trajectories that maximize alignment of warped events.Such methods support motion, camera-motion, depth, and segmentation tasks.
  • The paper introduces twenty two focus loss functions and connects event-alignment metrics to shape-from-focus and autofocus functions.The connection brings established image-analysis tools into event-based vision while retaining asynchronous timestamps and polarity.
  • The proposed taxonomy compares focus loss functions by accuracy and computational effort and demonstrates their use for depth and optical-flow estimation.The evaluation uses a recent publicly available dataset.

2. Event-based Camera Working Principle

Event cameras generate asynchronous, sparse space-time event streams rather than constant-rate images. Each event records where and when a brightness change crossed a threshold, together with its polarity.

  • A DVS event is generated when logarithmic brightness at a pixel changes by a threshold C, and it records the pixel location, timestamp, and polarity.The polarity indicates the sign of the brightness change.
  • Each pixel has its own input-dependent sampling rate, so event cameras output sparse asynchronous data instead of fixed-rate images.The output rate is proportional to the amount of motion in the scene.

3. Methodology

The method warps asynchronous events along parameterized point trajectories into an image of warped events, then optimizes focus loss functions to measure alignment. It extends variance-based alignment with twenty two image-based metrics spanning statistical, derivative, spatial, sharpness, dispersion, and focus-based formulations.

  • Motion Compensation Framework: Events are transported along parameterized point trajectories to a reference time, producing warped events whose accumulated image representation supports alignment analysis.The trajectory parameters encode motion or scene parameters, while polarity may be retained or ignored during accumulation.
  • Focus Loss Functions: The proposed taxonomy compares twenty two focus loss functions classified by statistical or derivative operators, spatial dependence, and optimization direction.The metrics connect event alignment with focus measures used in shape-from-focus and autofocus.
  • Variance Loss: The original framework maximizes IWE variance, interpreting stronger variance as greater contrast and stronger aligned edges.Variance measures pixel-value dispersion independently of spatial arrangement and corresponds to the AC, or high-frequency, component of the IWE.
  • Image Sharpness: Sharpness-based losses maximize high-frequency edge evidence using gradient, Hessian, Laplacian, Difference-of-Gaussians, or Laplacian-of-Gaussian responses.Derivative magnitudes assess high-frequency image content because derivative operators act as band-pass or high-pass filters.
  • Image Sharpness: Area-based alignment minimizes the IWE support, whose weighting functions assign contributions across pixel-value thresholds and whose support maps visualize local contributions.The support is designed to represent edge thickness: sharper IWEs have thinner edges and smaller effective area.
  • Image Dispersion and Focus Connection: Dispersion-based alternatives include a robust image-range measure and Moran’s I, while the focus principle also treats varying motion or scene parameters as slices of a focal stack.The proposed range formulation avoids brittle direct minimum and maximum statistics by using the support of the image probability distribution.

4. Experiments

The experiments compare focus losses primarily on rotational motion, with qualitative evaluations for depth and optical flow. Variance, derivative-based losses, and high-pass filters provide strong accuracy or speed, while implementation and polarity affect results.

  • Accuracy and timing evaluation: Rotational motion is the main accuracy benchmark because motion-capture ground truth is precise, whereas depth and optical-flow ground truth is less reliable.The benchmark uses one-minute sequences reaching approximately ±1000 °/s and processes more than 160 million events.
  • Accuracy and timing evaluation: Derivative-based, high-pass, and variance-based losses achieve less than 2.6% angular-velocity error, while entropy and support losses remain below 7%.Using polarity generally improves accuracy slightly.
  • Accuracy and timing evaluation: The variance loss is fastest at 17 µs, negligible beside the approximately 2 ms event-warping cost, and offers a notable accuracy–speed trade-off.Loss functions operate on the IWE with O(Np) complexity, while IWE construction has O(Ne) complexity.
  • Accuracy and timing evaluation: The gradient-magnitude evaluation reports approximately 18 °/s RMS error, or 2.7% of the maximum angular velocity.The result is shown across four 15-second subintervals and motion axes.
  • Depth and optical flow: Depth loss curves vary smoothly with depth and show extrema near the correct value, while optical-flow surfaces peak at the ground-truth flow and differ in sharpness.The Laplacian magnitude and its variance produce the narrowest optical-flow peaks.
  • Accuracy and timing evaluation: Runtime figures are illustrative because implementation was not optimized and GPU warping could provide considerable speedups.The reported timings use C++ Standard Library and OpenCV implementations on a single-core CPU.
  • Unsupervised optical flow: A neural network trained with the gradient-magnitude loss increases event alignment, but deeper optical-flow evaluation is left for future work.The network also uses a Charbonier prior on the flow derivative.

5. Conclusion

The paper concludes that focus losses connect event alignment with shape-from-focus and autofocus, enabling established image-processing analysis for event data. Across accuracy and runtime comparisons, variance, gradient magnitude, and Laplacian losses are among the strongest choices.

  • Contributions: The study compares twenty-two focus loss functions for event-based processing and categorizes them by edge strength, dispersion, and information content.The functions include metrics developed from basic principles, such as the area of warped-event images.
  • Connection to conventional imaging: Focus losses connect event alignment with shape-from-focus and autofocus, allowing mature computer-vision tools to be applied to asynchronous event data.The connection uses analogous focus scores over images or histograms.
  • Evaluation and conclusion: The comparison concludes that variance, gradient magnitude, and Laplacian losses are among the best functions across accuracy and computational time.The conclusion presents this as the paper’s overall comparative finding.
  • Focus metrics: The area metric measures edge thickness through a weighted treatment of level-set interiors rather than a brittle fixed pixel threshold.This design is introduced for edge-like images of warped events.

B.1. Definition of the Area of an Image

The area metric measures the support of a warped-event image across intensity thresholds, with weighting that avoids dependence on one arbitrary threshold. Minimizing this area favors alignment because coincident warped events produce thinner support.

  • Support definition: A meaningful support measure counts pixels whose warped-event image exceeds a threshold because Gaussian kernels otherwise give the image unbounded support.The threshold is represented through the Heaviside function.
  • Alignment principle: The area of a one-dimensional warped-event image is minimized when corresponding events are warped to the same location.For two events, smaller separation produces smaller support or edge thickness.
  • Threshold aggregation: Summing support over thresholds with a decreasing weighting function makes the metric less dependent on the chosen threshold and emphasizes low-threshold regions.Unit-area weighting gives the aggregate support the same units as an individual support measure.
  • Assumption: The support definition assumes nonnegative image values, or applies separately to positive and negative event images before adding the results.This assumption corresponds to omitting polarity or separating polarities.
  • Illustration: The image-area examples show that the better-aligned warped patch has thinner edges and smaller support.The illustrated support values are 0.72 |Ω| for the suboptimal patch and 0.65 |Ω| for the optimal patch.

B.2. Simplification of the Area of an Image

The area of an image of warped events can be computed directly from continuous pixel values, avoiding threshold-by-threshold evaluation. Support maps show how accumulated warped events contribute to this area, while entropy maximization favors high-contrast alignment.

  • Area simplification: The continuous formulation computes the image support directly from I(x), avoiding evaluation at every threshold λ and summation of partial results.Analytical integration replaces repeated threshold computations.
  • Support maps: Support-map contributions increase when pixels accumulate approximately λ0 or more warped events, producing higher per-pixel support.For λ0 = 10 warped events, pixels with I(x) ≳ λ0 contribute more than pixels with I(x) ≲ λ0.
  • Support maps: Red support-map regions approximate the IWE area, whereas blue regions mark pixels with few accumulated warped events.The image area defines the corresponding focus loss function.
  • Entropy: Maximizing IWE entropy favors large-magnitude pixel values created by aligned-event aggregation, increasing image range and contrast.The entropy argument favors low-density values away from I = 0.

D. Loss Function: Image Range

The image-range section develops focus measures based on the support of the IWE and related spatial statistics. These measures connect contrast, spatial dissimilarity, and edge strength to event alignment.

  • Spatial statistics: Moran’s local statistic is positive when a pixel and its neighbors lie on the same side of the mean and negative when they lie on opposite sides.Increasing event alignment corresponds to favoring negative local Moran indices.
  • Spatial statistics: Geary’s local statistic acts as an edge detector because large values identify regions with dissimilar intensities across object edges.Homogeneous regions have positive spatial autocorrelation with c(x) < 1.
  • Local statistics: Local mean absolute deviation is closely related to another edge-detecting local measure, differing by its L1 rather than L2 norm.Weighted neighborhoods permit an efficient approximation using two convolutions.
  • Local statistics: Local MAV does not provide enough information to estimate warp parameters when event polarity is not used.This limitation is stated for the polarity-agnostic setting.

G. Plots of the Local Loss Maps

The local-loss figures compare unoptimized and motion-compensated IWEs using a shared color scale. Their interpretation depends on whether the objective is maximized or minimized.

  • Figure layout: The figures show local-loss integrands before and after optimizing focus loss functions for dynamic and boxes scenes.The images are arranged in pairs, with the uncompensated result on the left and the motion-compensated result on the right.
  • Displayed losses: Area-based losses use support maps, whereas IWE range has no displayed integrand because it is not an image-domain integral.MAV and local MAV are also omitted because they cannot be optimized with respect to the parameters.
  • Color interpretation: For maximization objectives, better alignment appears as yellow regions, while for minimization objectives it appears as increasingly blue regions.All maps use a blue-to-yellow scale from small to large values.

H. Additional Experiments on Accuracy Evaluation

Additional experiments evaluate focus loss functions on rapidly rotating sequences from the referenced dataset. The reported RMS errors are small relative to the signal excursion.

  • Experimental setting: Tables 3 and 4 evaluate focus loss functions on boxes and poster sequences undergoing rotational velocities close to 1000 °/s.The sequences come from dataset [45].
  • Evaluation: The RMS columns of Tables 3 and 4 are summarized in Table 2.The summary consolidates the reported accuracy measurements.
  • Evaluation: The reported RMS errors are small compared with the excursion of the signal.This comparison is the stated interpretation of the additional quantitative evaluations.

I. Additional Plots of Focus Loss Functions in Optical Flow Space

Optical-flow experiments visualize how focus losses behave around candidate motion vectors, showing distinct effects of polarity and optimization direction. Variance and derivative-based losses form clear optima near the correct flow, while several alternatives are weaker or polarity-dependent.

  • Experimental setup: Approximately 2,000 events from a 31 × 31-pixel, 200 ms window were evaluated around three candidate flow vectors.Heat maps cover ±60 pixel/s around the ground-truth flow, approximately θ2 = (−40, 0) pixel/s.
  • Without Polarity: Without polarity, variance has a clear peak at the correct flow, whereas MS and MAD peaks are less pronounced and MAV misses the ground-truth location.The local versions are slightly narrower than their global counterparts.
  • Without Polarity: Without polarity, area-based losses show local minima at the correct flow, while range and Geary’s C peak there; entropy does not.Moran’s index instead has the expected local minimum at the correct flow.
  • With Polarity: With polarity, entropy improves and derivative-based losses produce clear, slightly more pronounced peaks at the correct flow.Polarity also makes the mean-square and variance-related relationships more apparent when positive and negative events are balanced.
  • With Polarity: Polarity makes MS approximately equal to variance and MAV approximately equal to MAD when positive and negative events are balanced.The area-based losses remain similar apart from vertical scaling, while Moran’s index and Geary’s C become brittle.

J. Additional Plots on Depth Estimation

Depth experiments show that focus curves vary smoothly with depth and attain extrema near the correct value, enabling semi-dense 3D reconstruction. Local MS and MAD provide good reconstructions, while curve width and confidence-map processing affect precision and noise.

  • Depth Estimation: Focus curves vary smoothly with depth and show clear extrema around the correct depth, with a sharp IWE near Z = 1.11 m.Narrower curves provide better properties for determining the optimal depth, whereas range curves have wide peaks and lower precision.
  • Semi-dense 3D Reconstruction: Semi-dense reconstruction selects each pixel’s depth from the peak of a focus curve computed on Gaussian-weighted 3 × 3 patches.The peak value is also recorded as a focus confidence map.
  • Semi-dense 3D Reconstruction: Adaptive thresholding of the focus confidence map and median filtering remove spike noise, with depth estimated most reliably at strong brightness edges.The resulting depth map is also visualized as a depth-colored point cloud.
  • Reconstruction quality: The tested methods generally produce good depth reconstructions, but differ in noise and edge recovery because of focus-confidence-map shapes and thresholding parameters.Local MS and local MAD produce good results despite their simple form.
  • Reconstruction quality: The reconstruction experiment represents depth from 0.45 m to 2.4 m using 1,000,000 events over 2.93 s and an 85 cm camera baseline.Depth-colored semi-dense maps and point clouds are shown for several focus losses.

K. Analytical Derivatives of Focus Loss Functions

The paper derives analytical derivatives for focus losses and the warped-event image, enabling gradient-based optimization with established calculus operations. It also describes smoothing and convolution-based computations that improve numerical robustness and efficiency.

  • Derivative framework: Analytical derivatives are provided for focus losses and enable faster, more accurate optimization than numerical derivatives.The losses are expressed through well-known analytical operations.
  • Derivative framework: The event-warping derivative is obtained with respect to θ by approximating the Dirac delta with a Gaussian and applying the chain rule.The parameter vector is θ = (θ1, . . . , θM)^⊤ ∈ R^M.
  • Statistical losses: Derivatives of MS, variance, MAV, MAD, entropy, area, range, and local statistics follow from the chain rule and linear operator properties.For centered or locally centered IWEs, means and derivatives commute because both are linear operators.
  • Numerical implementation: The MAV derivative can use tanh(kx) as a smooth approximation to sign(x), with k controlling the transition width.This replaces the nonsmooth sign operation for numerical computation.
  • Entropy: Entropy derivatives use a smoothed histogram approximation of the IWE density, with Gaussian filtering reducing discretization noise and improving robustness.A high-resolution normalized histogram is smoothed before derivative evaluation.
  • Derivative-based losses: Convolution derivatives are computed component-wise, supporting derivative expressions for gradient, Laplacian, Hessian, DoG, LoG, and related variance losses.The DoG derivative uses σ1 > σ2, while the DoG approximates the LoG when σ2 ≈ 1.6σ1.
Loading 1904.07235v1…