Source-linked AI summary
EV-IMO: Motion Segmentation Dataset and Learning Pipeline for Event Cameras
Anton Mitrokhin, Chengxi Ye, Cornelia Fermuller, Yiannis Aloimonos, Tobi Delbruck
TL;DR
Estimating object motion alongside scene reconstruction from event data remains largely unexplored. This paper introduces a compositional neural-network pipeline and EV-IMO dataset for event-based motion segmentation, and reports that the pipeline generalizes well across scenes.
Problem
Estimating object motion alongside spatio-temporal scene reconstruction remains largely unexplored, and prior work did not provide an evaluation.
Source: §I · p. 2Method
A compositional neural-network pipeline uses monocular event data for up-to-scale depth and pixel-wise motion segmentation, unsupervised 6 dof egomotion, and per-segment linear velocity estimation.
Source: §I · p. 2Results
The pipeline estimates camera egomotion, dense depth, and independently moving objects’ 3D linear velocities, and is reported to generalize well across scenes.
Source: §V · p. 6, §VI · p. 7Takeaways & Limitations
The pipeline can be used for indoor motion estimation and obstacle avoidance.
Source: §I · p. 2The 2D network input may suffer from motion blur during fast motions; the pipeline addresses this with a fine-scale warping loss using 1 ms slices.
Source: A. Network Input · p. 3
Abstract
from arXiv · showhide
We present the first event-based learning approach for motion segmentation in indoor scenes and the first event-based dataset - EV-IMO - which includes accurate pixel-wise motion masks, egomotion and ground truth depth. Our approach is based on an efficient implementation of the SfM learning pipeline using a low parameter neural network architecture on event data. In addition to camera egomotion and a dense depth map, the network estimates pixel-wise independently moving object segmentation and computes per-object 3D translational velocities for moving objects. We also train a shallow network with just 40k parameters, which is able to compute depth and egomotion. Our EV-IMO dataset features 32 minutes of indoor recording with up to 3 fast moving objects simultaneously in the camera field of view. The objects and the camera are tracked by the VICON motion capture system. By 3D scanning the room and the objects, accurate depth map ground truth and pixel-wise object masks are obtained, which are reliable even in poor lighting conditions and during fast motion. We then train and evaluate our learning pipeline on EV-IMO and demonstrate that our approach far surpasses its rivals and is well suited for scene constrained robotics applications.
I. INTRODUCTION
The paper targets event-based motion understanding for robotics, where moving objects and camera motion must be estimated together. It introduces a neural pipeline and EV-IMO, a dataset with ground-truth motion and depth annotations.
I. INTRODUCTION
- Mobile robots need to detect moving objects, estimate their distance and trajectories, and remain aware of egomotion; event sensors offer high temporal resolution, low latency, and tolerance to poor lighting.Event streams encode motion of image contours and can capture extremely fast motion without motion blur.Source: §I · p. 1
- The figure illustrates ground-truth depth and pose alongside network predictions on EV-IMO; pose colors indicate object linear velocities, while camera egomotion is not visualized.The top row shows ground truth and the bottom row shows predicted output.Source: §I · p. 1
- Estimating object motion alongside spatio-temporal scene reconstruction remains largely unexplored in learning-based work, and an exception lacks evaluation.Classical reconstruction approaches are computationally heavy and can fail in the presence of outliers.Source: §I · pp. 1–2
- The compositional neural pipeline uses monocular event data to estimate up-to-scale depth, pixel-wise motion segmentation, unsupervised 6 dof egomotion, and per-segment linear velocity.The pipeline is intended for indoor motion estimation and obstacle avoidance.Source: §I · p. 2
- EV-IMO provides 32 minutes of indoor recordings and, to the authors’ knowledge, the first event-camera dataset with accurate pixel-wise moving-object masks alongside depth and trajectory ground truth.The objects and camera motions vary across backgrounds; the dataset also includes camera and object motion ground truth.Source: §I · p. 2
- The contributions also include an event-alignment loss based on motion-compensated event sharpness and a shallow multi-level network that retains similar performance to the full-sized network.The network estimates camera and object 3D motion from event data.Source: §I · p. 2
II. RELATED WORK
Event-based work spans optical-flow methods and 3D motion estimation, while the latter has been studied under several visual-odometry and sensor configurations.
II. RELATED WORK
- Event-based optical-flow methods use gradient computation,,,, template matching, and frequency computation, on event surfaces.Zhu et al. [45] use intensity from a DAVIS camera for self-supervised deep learning.Source: A. Event based Optical Flow, Depth and… · p. 2
- 3D motion estimation has been studied for rotation-only and known-map settings, and with image measurements or IMU sensors.Other approaches jointly reconstruct scene intensity and estimate 3D motion, considering rotation-only and general motion.Source: A. Event based Optical Flow, Depth and… · p. 2
B. Independent Motion Detection
Prior motion segmentation includes 2D and 3D geometric approaches and learning-based methods; this work's context also includes event-based SfM and event-slice inputs.
B. Independent Motion Detection
- 3D motion segmentation constrains flow through rigid camera motion and combines camera constraints with scene constraints for clustering regions.Methods include epipolar contradictions, subspace constraints, planar or multi-plane models, scene-complexity-based models, and occlusion and discontinuity constraints,.Source: B. Independent Motion Detection · p. 2
C. Learning in Structure from Motion
- SfM-Net is the closest related architecture, but its object-motion estimation was not evaluated for lack of a dataset; this work adds event-based object-motion evaluation.The cited event-based approaches focus on optical flow or full SfM and do not both exploit event-cloud structure and evaluate independently moving objects.Source: C. Learning in Structure from Motion · p. 2
A. Network Input
- The network receives consecutive 25 ms event slices projected into maps of positive counts, negative counts, and aggregated timestamps.This 2D representation reduces sparsity and improves efficiency over 3D learning, while a 1 ms fine-scale warping loss addresses possible motion blur.Source: A. Network Input · p. 3
B. Overview of the Architecture
The pipeline pairs depth and pose networks to derive optical flow and train through depth, mask, and warping losses. Its ego-motion model links inverse depth and flow to 3D motion, with depth and translation recovered up to scale.
B. Overview of the Architecture
- The depth network processes one event-slice map, while the pose network uses up to five consecutive maps to predict pixel-wise 3D motion and motion masks.Depth and masks receive supervision; the two network outputs generate optical flow, and inversely warped event slices are trained using warping quality.Source: B. Overview of the Architecture · p. 3
C. Ego-motion Model
- Given inverse depth, each pixel's optical flow is linearly related to 3D motion parameters, but depth and translation are determined only up to scale.The camera is assumed calibrated and to move rigidly; the implementation normalizes depth by average depth.Source: C. Ego-motion Model · p. 3
- Each object's motion is modeled as rigid background motion plus a 3D translation, with pixel-wise mask weights combining object translations.The object translations are modeled without rotation because most objects are relatively small.Source: C. Ego-motion Model · pp. 3–4
D. A Mixture Model for Ego-motion and Independently Moving Objects
A mixture model combines camera motion and object translations into pixel-wise poses, while event-warping losses train alignment at coarse and fine temporal scales.
D. A Mixture Model for Ego-motion and Independently Moving Objects
- Soft pixel-wise mixture weights combine ego-motion and residual object translations into candidate poses, allowing pixels to represent multiple rigid motions.The authors report that soft assignment better explains motion directions, while supervised masks yield qualitatively sharp object boundaries.Source: D. A Mixture Model for Ego-motion and… · p. 4
- The mixture representation can distinguish object regions whose 3D motions differ only slightly.Source: D. A Mixture Model for Ego-motion and… · p. 4
1) Event Warping Loss:
- Warping loss measures motion compensation first between event slices and then within slices using event timestamps.The coarse stage uses 3–5 consecutive 25 ms slices and compares warped event counts; the fine stage subdivides slices into 1 ms intervals to use continuous-stream timing.Source: 1) Event Warping Loss · p. 4
E. Loss functions
- At fine scale, the quasi-norm of stacked inversely warped event slices favors a sharp profile over a blurred one.The losses are computed at multiple network scales and weighted by pixel count before summation.Source: 1) Event Warping Loss · p. 4, E. Loss functions · p. 4
2) Motion Mask Loss:
The motion-mask loss uses ground-truth masks to assign ego-motion to background pixels and encourages locally smooth mixture weights. The depth loss matches predictions to ground truth and regularizes depth smoothness.
2) Motion Mask Loss:
- Ground-truth motion masks supervise the mixture weight for the ego-motion pose component in background regions.A separate smoothness loss penalizes first-order gradients of all mixture weights.Source: 2) Motion Mask Loss · p. 4
3) Depth Loss:
- Ground-truth depth supervises the depth output after prediction and truth are adjusted to the same scale.The loss also applies a smoothness penalty to second-order gradients of predicted depth.Source: 3) Depth Loss · p. 4
F. Evenly Cascading Network Architecture
The network uses an evenly cascading encoder-decoder to aggregate features and refine predictions across resolutions. The dataset pipeline combines 3D scans with VICON tracking to generate depth and object masks at high temporal resolution.
F. Evenly Cascading Network Architecture
- The evenly cascading network aggregates multilevel feature streams, improving low-level features through bilinear scaling and residual learning while building high-level features.Its encoder-decoder connects same-shape features with U-Net-style skip links.Source: F. Evenly Cascading Network Architecture · p. 4
G. Prediction of Depth and Component Weights
- Decoding begins with a coarse backbone prediction, then upsamples and merges features to refine the map through successive prediction-residue additions.High-level and encoding features modulate enhanced lower-level features used to estimate the residue.Source: G. Prediction of Depth and Component… · pp. 4–5
IV. EV-IMO DATASET
- The dataset is the first event-camera dataset described here with multiple independently moving objects, camera motion, and accurate depth, masks, and trajectories above 200 frames per second.The labeling pipeline produced more than 30 sequences totaling half an hour.Source: §IV · p. 5
1) Methodology:
- The pipeline scans objects and the room before recording, tracks objects and camera with VICON, then projects transformed point clouds to produce pixel masks and ground-truth depth.Interpolating VICON poses allows ground truth at virtually any event timestamp; the approach requires expensive equipment.Source: 1) Methodology · p. 5
2) Dataset Generation:
Dataset generation aligns scanned objects and room geometry with VICON tracking, then projects the geometry into the camera view to create depth and mask ground truth. The dataset also supplies camera and object velocities for evaluation.
2) Dataset Generation:
- Objects are 3D-scanned with VICON reflective markers, and RANSAC locates marker positions to transform point clouds into the world frame.Room scanning uses an Asus Xtion RGB-D sensor, with tracking used to initialize global ICP alignment.Source: 2) Dataset Generation · p. 5
- The camera center is calibrated in the world frame using a wand tracked by both VICON and the camera, then the static point cloud is projected to camera pixels.The projection uses the camera matrix and transformations among the camera, markers, world, and point cloud.Source: 2) Dataset Generation · p. 5
- The dataset provides high-resolution depth, pixel-wise object masks, camera and object trajectories, and instantaneous velocities in the camera frame.Ground-truth frame rates are configurable; experiments used 40 frames per second.Source: 2) Dataset Generation · p. 5
A. Sequences
EV-IMO sequences vary in background, texture, object count, speed, and lighting, with motion diversity emphasized for event-camera data. Experiments test on held-out background types and show that the 40k-parameter network retains reasonable depth, egomotion, and masks, though it does not reliably predict object velocity.
A. Sequences
- EV-IMO sequences vary across background depth and texture, moving-object count, motion speed, and lighting conditions.The dataset groups backgrounds as table, boxes, plain wall, and floor, and also includes cluttered tabletop scenes.Source: A. Sequences · p. 5
V. EXPERIMENTS
- The network estimates camera egomotion, dense depth, and independently moving objects’ 3D linear velocities despite extremely sparse event data.The evaluation trains on box and floor backgrounds and tests on table and plain-wall backgrounds; object masks are thresholded at 0.5 for IoU.Source: §V · p. 6
- The 40k-parameter network produces reasonable egomotion, depth, and motion masks, but does not reliably predict object velocity.Object translational velocities can instead be extracted by tracking the masks.Source: §V · p. 6
1) Qualitative Evaluation:
The evaluation presents qualitative predictions for depth, pixel-wise pose, and motion masks, and examines segmentation and motion estimation across network sizes and comparisons with prior work.
1) Qualitative Evaluation:
- Predicted masks and pixel-wise pose are shown across scenes with differing motion, object counts, and texture, alongside depth outputs and ground truth.Fig. 6 orders columns as DVS input, depth ground truth and output, pixel-wise pose ground truth and prediction, then predicted motion mask.Source: 1) Qualitative Evaluation · p. 6, Figure 6 · p. 7
- Reducing network size affects background depth only slightly, while object-mask and depth quality suffer notably.Source: 1) Qualitative Evaluation · p. 6
2) Segmentation and Motion Estimation:
- Motion evaluation uses AEE for linear velocities, RRE for camera rotation, and IoU for segmentation; Table II includes 40k-network values in braces.Monocular evaluations apply ground-truth scale; AEE is reported in m/s and RRE in radians/s.Source: 2) Segmentation and Motion Estimation · p. 6, Table II · p. 6
3) Comparison With Previous Work:
- The [24] egomotion comparison is restricted to sequences without independently moving objects or roll/pitch motion and with planar backgrounds; it yields no meaningful results on high-depth-variation scenes.Table III compares EV-IMO with [24], reporting translation in m/s and rotation in rad/s.Source: 3) Comparison With Previous Work · pp. 6–7, Table III · p. 7
- ECN [41] was compared despite not being designed for independently moving-object segmentation, and its relatively low performance is attributed to EV-IMO’s more complex motion and diverse depth.The evaluation included varied backgrounds and textures, with texture absence noted as limiting for event-based sensors.Source: 3) Comparison With Previous Work · p. 7
VI. CONCLUSIONS
The paper concludes that its compositional neural-network pipeline generalizes across different scenes and presents EV-IMO as an indoor dataset captured with VICON. It identifies event-stream augmentation, temporal tracking, and object grouping as future research directions.
VI. CONCLUSIONS
- The compositional neural-network pipeline combines unsupervised and supervised components and is reported to generalize well across different scenes.Source: §VI · p. 7
- The paper presents an event-based motion-segmentation method evaluating both camera and object motion, alongside the indoor EV-IMO dataset recorded using VICON.Source: §VI · p. 7
- Future work targets event-stream augmentation, estimation linked across successive time slices, and alternatives for grouping objects in the pipeline.The passage specifies partially or fully simulated data for event-stream augmentation.Source: §VI · pp. 7–8