PaperlayerView exact revision on arXiv

Source-linked AI summary

EVDodgeNet: Deep Dynamic Obstacle Dodging with Event Cameras

Nitin J. Sanket, Chethan M. Parameshwara, Chahat Deep Singh, Ashwin V. Kuruttukulam, Cornelia Fermüller, Davide Scaramuzza, Yiannis Aloimonos

arXiv:1906.02919v3cs.ROcs.CV

TL;DR

The paper addresses dynamic obstacle avoidance on quadrotors using event cameras and deep learning. It combines event-frame deblurring with networks for segmentation and optical flow, then evaluates the approach on a real quadrotor across varied scenarios. The reported component evaluations find improved generalization from deblurring and greater robustness of deblurred homography estimates across integration times.

  • Problem

    Dynamic-scene datasets often omit moving objects, while conventional camera-based deep-learning methods may be high-latency, computationally expensive, or fail to generalize to novel objects without retraining.

    Source: §I · p. 1, S.VII. MOVING OBJECT DATASET (MOD) · p. 11

  • Method

    The approach uses an encoder-decoder network to deblur event frames and an ego-motion-warped input to perform independently moving object segmentation and optical flow.

    Source: A. EVDeBlurNet · p. 3, B. EVSegFlowNet · p. 4

  • Results

    Deblurring improves generalization to novel objects, and deblurred homography estimation is more robust to changes in integration time.

    Source: A. Experimental Results and Discussion · pp. 6–7

  • Takeaways & Limitations

    The paper reports that networks trained in simulation transfer directly to real scenes without fine-tuning or retraining.

    Source: §V · p. 7

Abstract

from arXiv · show

Dynamic obstacle avoidance on quadrotors requires low latency. A class of sensors that are particularly suitable for such scenarios are event cameras. In this paper, we present a deep learning -- based solution for dodging multiple dynamic obstacles on a quadrotor with a single event camera and on-board computation. Our approach uses a series of shallow neural networks for estimating both the ego-motion and the motion of independently moving objects. The networks are trained in simulation and directly transfer to the real world without any fine-tuning or retraining. We successfully evaluate and demonstrate the proposed approach in many real-world experiments with obstacles of different shapes and sizes, achieving an overall success rate of 70% including objects of unknown shape and a low light testing scenario. To our knowledge, this is the first deep learning -- based solution to the problem of dynamic obstacle avoidance using event cameras on a quadrotor. Finally, we also extend our work to the pursuit task by merely reversing the control policy, proving that our navigation stack can cater to different scenarios.

I. INTRODUCTION AND PHILOSOPHY

Low-latency perception matters for agile small aerial robots, while event cameras provide high-temporal-resolution sensing for dynamic obstacle avoidance. The paper proposes an onboard deep-learning framework for dodging multiple dynamic obstacles and extends it to pursuit.

I. INTRODUCTION AND PHILOSOPHY

  • Event cameras output per-pixel temporal intensity differences caused by relative motion, with microsecond latency and high temporal resolution.The paper defines latency as the time to perceive, interpret, and generate control commands.Source: §I · p. 1

A. Problem Formulation and Contributions

B. Related Work

Related work spans ego-motion and independent-motion estimation, image-stabilization-based segmentation, and aerial-robot obstacle avoidance. The paper organizes perception and control into a navigation stack that includes EVDeBlurNet for event-frame deblurring.

1) Independent Motion Detection and Ego-motion Estimation – Two sides of the same coin:

  • Prior ego-motion work often focuses on static scenes and rejects moving objects as outliers, although modeling those outliers can estimate both ego-motion and independent-object motion.Source: 1) Independent Motion Detection and… · p. 2

2) Image stabilization as a key to independent motion segmentation:

3) Obstacle avoidance on aerial robots:

C. Organization of the paper

  • The paper divides its navigation stack into perception—event-frame deblurring, ego-motion estimation, and IMO segmentation and optical flow—and a control module for dodging.Source: C. Organization of the paper · p. 2

II. DEEP LEARNING BASED NAVIGATION STACK FOR DODGING DYNAMIC OBSTACLES

A. EVDeBlurNet

  • EVDeBlurNet addresses event-frame misalignment by balancing contrast against similarity to the input, producing deblurred frames with an encoder-decoder network.Its lossy reconstruction also removes stray events and retains contour events, increasing signal-to-noise ratio.Source: A. EVDeBlurNet · p. 3
  • Unlike a prior method that couples deblurring to optical-flow prediction, this work presents problem-independent deblurring without optical-flow supervision [23].Source: A. EVDeBlurNet · p. 3

B. EVSegFlowNet

EVSegFlowNet jointly learns IMO segmentation and optical flow, avoiding two separate networks while targeting more accurate foreground motion estimates. Its inputs use ego-motion-warped event frames, and the networks transfer from simulation to real scenes without retraining or fine-tuning.

B. EVSegFlowNet

  • Unsupervised optical flow can estimate background motion accurately but struggles on foreground pixels because the foreground is often much smaller than the background.The authors note that many training iterations risk overfitting while trying to improve foreground flow.Source: B. EVSegFlowNet · p. 4
  • EVSegFlowNet learns IMO segmentation and optical flow together in one semi-supervised network, addressing complexity and accuracy concerns with using separate networks.Source: B. EVSegFlowNet · p. 4
  • Ego-motion-warped event frames align background pixels, leaving IMO misalignment for the network to learn as sub-pixel-accurate optical flow.Source: B. EVSegFlowNet · p. 4
  • All networks are trained in simulation and transfer directly to the real world without retraining or fine-tuning.Source: B. EVSegFlowNet · p. 4

III. CONTROL POLICY FOR DODGING IMOS

The control policy addresses multiple known and unknown independently moving objects, using different strategies for three levels of object-shape information. For known-radius spheres, it estimates motion and selects a safe direction for the quadrotor to evade.

III. CONTROL POLICY FOR DODGING IMOS

  • The policy distinguishes known-radius spheres, unknown shapes with a size bound, and objects with no prior information; geometric priors support more robust results and finer control.Source: §III · p. 4

A. Sphere with known radius r

  • For a sphere under gravity, the method estimates its initial 3D position from its known radius, tracks it across event frames to estimate velocity, and predicts its trajectory.Source: A. Sphere with known radius r · p. 5
  • For a single known-radius sphere, the safe direction is opposite the vector from the quadrotor to the projected trajectory intersection, and the PID controller moves at least the extended obstacle size.Source: A. Sphere with known radius r · p. 5
  • For multiple spherical objects, the method assumes no mutual occlusion in the front event-camera frame, clusters each object, and estimates each object's 3D position and velocity.Source: A. Sphere with known radius r · p. 5

B. Unknown objects with no prior knowledge

Unknown-object evasion chooses a safe image-plane direction perpendicular to the object's motion, since monocular sensing cannot recover its 3D velocity without prior knowledge. Experiments report successful dodging and pursuit across varied obstacles, alongside evaluations of deblurring, homography estimation, and segmentation.

B. Unknown objects with no prior knowledge

  • Without prior object knowledge, monocular vision cannot determine an IMO's 3D velocity, so the quadrotor selects an image-plane direction perpendicular to its motion.For multiple objects, the method adapts the cyclic-pair strategy using image-plane velocity vectors.Source: B. Unknown objects with no prior… · p. 5

A. Experimental Results and Discussion

  • Success rates were 86% for a ball and car, 76% for an airplane and synchronized multiple objects, 83% for both Bebop 2 tasks, and 70% in low light.Dodging success required detection and evasion; pursuit success required detection and collision.Source: A. Experimental Results and Discussion · p. 5
  • Five consecutive detections estimate object trajectory or image-plane velocity, yielding a 60 ms perception response lag before PID-controlled movement.Each frame pair takes 10 ms for neural-network computation and 2 ms for post-processing.Source: A. Experimental Results and Discussion · p. 6
  • D2 deblurring and denoising deteriorate at longer integration times, whereas D3 suppresses weak contours and works better at longer times.With D3, output is nearly absent at smaller integration times; Fig. 6 compares raw frames and D2/D3 outputs at 1, 5, and 10 ms.Source: A. Experimental Results and Discussion · p. 6, Figure 6 · p. 6
  • Deblurred homography estimates are more robust across integration times, while supervised homography variants outperform their unsupervised counterparts on both in-domain and novel textures.The authors speculate that sparse data and image-based similarity metrics poorly suit event frames.Source: A. Experimental Results and Discussion · pp. 6–7, Figure 7 · p. 7
  • Deblurring improves segmentation generalization to novel objects, while homography warping yields better results but only marginal segmentation gains in 3D scenes.The evaluation uses detection rate with threshold τ = 0.5.Source: A. Experimental Results and Discussion · p. 7, Table II · p. 7

V. CONCLUSIONS

The paper presents a shallow-network design for low-latency quadrotor navigation that dodges dynamic obstacles using a monocular event camera and onboard sensing. It reports simulation-to-real transfer without fine-tuning and extends the navigation stack to pursuit.

V. CONCLUSIONS

  • The navigation stack was extended to pursuit, and its control policy can be re-purposed for related tasks.The authors identify better event-frame similarity scoring or more robust event-frame construction as possible ways to improve results.Source: §V · p. 7
  • The shallow networks trained in simulation transfer directly to the real world without fine-tuning or retraining.Source: §V · p. 7
  • The approach uses reusable shallow neural-network building blocks for dynamic-obstacle dodging with a monocular event camera and onboard sensing.The design accounts for computation and latency requirements on micro/nano quadrotors.Source: §V · p. 7
  • To the authors’ knowledge, this is the first deep-learning-based solution to dynamic-obstacle avoidance using event cameras on a quadrotor.Source: §V · p. 7

S.I. DEFINITIONS OF COORDINATE FRAMES USED

The appendix defines the sensor coordinate frames and event-frame representation used in the pipeline. Event frames aggregate asynchronous triggers over a temporal window, providing a near-constant processing rate while retaining spatial, polarity, and timing information.

S.I. DEFINITIONS OF COORDINATE FRAMES USED

  • The IMU, front and down-facing event cameras, down-facing sonar, and world use frames I, EF, ED, S, and W, respectively; sensors are rigidly attached and calibrated.The image formation uses a pinhole model, and image-plane points are undistorted unless stated otherwise.Source: S.I. DEFINITIONS OF COORDINATE FRAMES… · p. 9

S.II. EVENT FRAME E

  • An event frame E collects triggers in a spatiotemporal window beginning at t0 with temporal depth δt.The three-channel representation stacks positive-event averages, negative-event averages, and average trigger times per pixel.Source: S.II. EVENT FRAME E · p. 9
  • Event frames are produced at a near-constant rate, allowing control commands within a constant time bound while preserving spatial relationships, polarity, and timing for the CNNs.The representation is used in place of processing each raw event individually.Source: S.II. EVENT FRAME E · p. 9

S.III. EVHOMOGRAPHYNET

EVHomographyNet estimates the transformation between successive event frames for down-camera odometry, with supervised and unsupervised training variants. The appendix also specifies contrast and distance-loss variants used in the method.

S.III. EVHOMOGRAPHYNET

  • The event-camera homography approach targets embedded, high-speed odometry from a down-facing camera over a planar surface, with speed–accuracy tradeoffs adjustable through network size.The paper presents this as its first deep-learning solution to homography estimation using event cameras.Source: S.III. EVHOMOGRAPHYNET · p. 10
  • The supervised variant trains on synthetic homography-warped event frames using predicted and ground-truth 4-point homographies.The unsupervised variant uses TensorDLT and a Spatial Transformer Network with bilinear interpolation and an image-similarity distance between event frames.Source: S.III. EVHOMOGRAPHYNET · p. 10

S.IV. LOSS FUNCTIONS

  • The contrast-function variants C1 and C2 use variance on image gradients rather than raw values, which empirically improved results and training stability relative to.The gradient operator is 2D and uses Sobel in this work.Source: S.IV. LOSS FUNCTIONS · p. 10
  • The event-frame distance variants are D1, D2, and D3: generic l1 photometric loss, Charbonnier loss, and the robust loss, respectively.For D3, α is output by the network.Source: S.IV. LOSS FUNCTIONS · p. 10

S.V. NETWORK DETAILS

The network details emphasize simple shallow architectures and describe training and input normalization. The compression analysis argues that the joint segmentation-flow network can reduce complexity by estimating flow only for foreground pixels and obtaining segmentation within the same network.

S.V. NETWORK DETAILS

  • The proposed networks use shallow architectures, with performance attributed to stacking multiple shallow networks; more advanced architectures are left for future work.The architecture diagram identifies EVDeblurNet, EVHomographyNet, and EVSegFlowNet and distinguishes convolution, deconvolution, and dropout blocks.Source: S.V. NETWORK DETAILS · p. 10, Figure S2 · p. 11
  • EVDeblurNet trains for 200 epochs at learning rate 10^-3, while EVSegNet, EVFlowNet, and EVSegFlowNet train for 50 epochs and EVHomographyNet for 200 epochs at learning rate 10^-4.The batch sizes are 256 except EVDeblurNet with D3, which uses 32; EVDeblurNet uses 256 for D1/D2.Source: S.V. NETWORK DETAILS · p. 10
  • All networks normalize event-frame pixels to [−1, 1]; for D3, networks output α in their last n channels, with N = 2n.Source: S.V. NETWORK DETAILS · pp. 10–11

S.VI. COMPRESSION ACHIEVED BY USING

  • EVSegFlowNet estimates flow only for foreground pixels and obtains segmentation from the same network, yielding the stated complexity relation O(eS) + O(eF) ≪ O(S) + O(F).The analysis defines complexity as the minimum number of neurons needed for satisfactory generalization and assumes near-linear scaling for smaller shallow networks.Source: S.VI. COMPRESSION ACHIEVED BY USING · p. 11

S.VII. MOVING OBJECT DATASET (MOD)

The Moving Object Dataset addresses the scarcity of visual odometry and SLAM datasets that include dynamic objects. It uses varied synthetic scenes and moving objects to generate training data, alongside a separate dataset for learning event-frame deblurring.

S.VII. MOVING OBJECT DATASET (MOD)

  • Existing visual odometry and SLAM datasets commonly assume static scenes, often excluding moving objects.The authors use synthetic scenes with moving objects to generate training data.Source: S.VII. MOVING OBJECT DATASET (MOD) · p. 11
  • Seven randomized configurations provide training data, with one reserved for testing generalization to novel scenes.Each configuration contains three moving objects; rendered images are converted to events at three integration times.Source: S.VII. MOVING OBJECT DATASET (MOD) · p. 11, Figure S3 · p. 12

A. 3D room and moving objects

  • The simulated rooms vary wall textures and lighting, while camera trajectories cover combinations of rotation and translation relevant to quadrotor motion.The rooms contain seven light sources for uniform illumination.Source: A. 3D room and moving objects · pp. 11–12
  • Three independently moving objects per room span varied shapes, textures, colors, sizes, and trajectories, moving ten times faster than the camera.The objects range from simple shapes to complex ones, and their trajectories produce varied relative poses.Source: A. 3D room and moving objects · p. 12, Figure S4 · p. 12, Figure S5 · p. 12

B. Dataset for EVDeblurNet

  • EVDeblurNet training uses 15K event frames from five floor textures and integration times of 1, 5, and 10 ms.The data comes from a downward-facing camera viewing planar textures without moving objects; training uses random 128 × 128 crops.Source: B. Dataset for EVDeblurNet · p. 12

C. Dataset for EVSegNet, EVFlowNet and EVSegFlowNet

This section describes event-frame datasets for segmentation, flow, and homography networks, including in-dataset and novel-scene testing. It also outlines a conservative evasion strategy for arbitrary-shaped objects when their size is bounded.

C. Dataset for EVSegNet, EVFlowNet and EVSegFlowNet

  • 70K event frames support segmentation and flow training, with 60K images for training and 10K for testing across six scenarios.Training uses two randomly selected integration times per scenario; testing uses 1 ms data because higher integration times have mask alignment errors.Source: C. Dataset for EVSegNet, EVFlowNet and… · p. 12

COMPARSION OF HOMOGRAPHY NETWORK PERFORMANCE FOR EVENT

  • Out-of-dataset testing uses 6K frames from a different room with realistic wall textures and immovable objects, testing generalization beyond training appearances.The test uses integration times of 1 and 2 ms; the authors designed it to assess whether the network learns object contours and motion information.Source: COMPARSION OF HOMOGRAPHY NETWORK… · pp. 13–14

D. Dataset for EVHomographyNet

  • EVHomographyNet training uses one frame at a time, with random 128 × 128 patches and randomly perturbed corners to create homography-warped event frames.The in-dataset test applies random perturbations to center crops, while out-of-dataset testing follows the EVSegFlowNet test setup.Source: D. Dataset for EVHomographyNet · p. 14
  • For an arbitrary-shaped object with a size bound, the method uses the projection’s convex hull and then assumes a sphere whose diameter is the maximum principal-axis bound.The sphere-based approach is more conservative and constrains sensing range and latency according to how close the bound is to the object’s actual size.Source: D. Dataset for EVHomographyNet · p. 14

S.IX. PURSUIT: A REVERSAL OF EVASION?

The paper demonstrates pursuit by reversing its evasion control policy and describes the indoor flight setup used for experiments. Its homography comparison reports similar best performance for event and RGB inputs despite a parameter-count difference, with relative performance depending on perturbation size.

S.IX. PURSUIT: A REVERSAL OF EVASION?

  • Pursuit is demonstrated by reversing the control policy for the presented evasion cases.For self-propelled objects such as quadrotors, the paper assumes a linear motion model and leaves intent prediction to future work.Source: S.IX. PURSUIT: A REVERSAL OF EVASION? · p. 14

S.X. EXPERIMENTAL SETUP

  • Indoor experiments used objects thrown or flown toward a hovering or slowly drifting quadrotor at 4.4–6.8 m/s from 3.6–5.2 m.The flight space measured about 6 × 5.5 × 3.5 m³, and textured carpets were laid down to support homography estimation.Source: S.X. EXPERIMENTAL SETUP · pp. 14–15

USING EVENT AND RGB FRAMES

  • Best event-frame and RGB homography networks perform almost similarly, although the event network uses fewer parameters.The event network uses 1.3 million parameters; the minimum reported for reasonable RGB results is 9.7 million.Source: USING EVENT AND RGB FRAMES · p. 15
  • RGB homography estimation works better for small perturbations, while event-frame contour information is more important for large perturbations.The RGB training and test images lacked motion blur; the authors expect real-world motion blur to degrade RGB-network performance unless RGB frames are deblurred.Source: USING EVENT AND RGB FRAMES · p. 15

Related papers

Loading 1906.02919v3…