Source-linked AI summary
Event-Based Motion Segmentation by Motion Compensation
Timo Stoffregen, Guillermo Gallego, Tom Drummond, Lindsay Kleeman, Davide Scaramuzza
TL;DR
Event-based cameras make motion acquisition attractive but complicate per-event separation of independently moving objects from camera-induced motion. The paper jointly estimates event-cluster associations and motion parameters through motion compensation, achieving up to 10% improvement over the state of the art. The method also provides quantitative evidence of event-based motion segmentation performance.
Problem
Per-event segmentation must distinguish independently moving objects and background from camera ego-motion, although each event carries little information.
Method
The method jointly estimates event-cluster associations and parametric cluster motions by alternating optimization of a motion-compensation-based objective.
Results
The method outperforms two recent methods on a public dataset by as much as 10% and resolves small relative motion differences between clusters.
Takeaways & Limitations
Avoiding explicit optical-flow estimation enables motion segmentation in challenging high-speed and HDR scenes while producing sharp motion-compensated object images.
Takeaways & Limitations
The alternating method converges only locally and requires initialization of θ and P, with no guarantee of reaching a global solution.
Abstract
from arXiv · showhide
In contrast to traditional cameras, whose pixels have a common exposure time, event-based cameras are novel bio-inspired sensors whose pixels work independently and asynchronously output intensity changes (called "events"), with microsecond resolution. Since events are caused by the apparent motion of objects, event-based cameras sample visual information based on the scene dynamics and are, therefore, a more natural fit than traditional cameras to acquire motion, especially at high speeds, where traditional cameras suffer from motion blur. However, distinguishing between events caused by different moving objects and by the camera's ego-motion is a challenging task. We present the first per-event segmentation method for splitting a scene into independently moving objects. Our method jointly estimates the event-object associations (i.e., segmentation) and the motion parameters of the objects (or the background) by maximization of an objective function, which builds upon recent results on event-based motion-compensation. We provide a thorough evaluation of our method on a public dataset, outperforming the state-of-the-art by as much as 10%. We also show the first quantitative evaluation of a segmentation algorithm for event cameras, yielding around 90% accuracy at 4 pixels relative displacement.
Supplementary Material
The supplementary material adds experiments and theory beyond the paper’s main presentation.
- The authors encourage readers to view the accompanying video and consult the supplement for added experiments and theory.
1. Introduction
The paper addresses per-event motion segmentation for event-based cameras, where camera ego-motion and independently moving objects create difficult classification problems. It proposes jointly estimating event-cluster associations and parametric motions through motion compensation, avoiding explicit optical-flow computation.
- Event-based cameras asynchronously record intensity changes at pixel locations with microsecond timestamps and polarity, sampling scenes according to their dynamics.
- Motion segmentation assigns events to independently moving objects, but camera ego-motion causes events across the image plane and makes per-event classification challenging.
- The method alternates between estimating event-cluster associations and cluster motion parameters using an objective function based on motion-compensated event-image sharpness.
- The method supports different parametric motions, including translation, rotation, and zooming, while segmenting events into clusters representing moving objects or background.
- 10% improvement over the state-of-the-art was reported in challenging high-speed and difficult-illumination scenarios, alongside accuracy in resolving small motion differences.
2. Related Work
Prior event-based segmentation methods use optical flow, learning, or sequential motion compensation, while this method performs joint per-event segmentation with flexible motion models and a single objective.
- Earlier methods detected or segmented objects using Hough transforms, optical-flow information, event corners, or learned separation of camera and object motion.
- Motion-compensation approaches fitted models greedily to dominant events and then analyzed remaining events for inconsistencies.
- The proposed method performs per-event segmentation rather than only producing bounding boxes for detected object regions.
- It supports general parametric motion models, optimizes a single objective, and handles occlusions between objects at any time.
- Figure 2 represents each cluster with a motion-compensated warped-event image and merges the cluster images using distinct colors.
- The method jointly estimates all objects’ apparent motions and event-object associations through iterative alternating optimization in an expectation-maximization fashion.
3. Methodology
The method segments event packets into motion-coherent clusters by jointly estimating event associations and motion parameters through alternating optimization of motion-compensated image sharpness. It avoids explicit optical-flow computation and produces motion-compensated appearance patterns for each cluster, while requiring initialization and a manually selected cluster count.
- Problem formulation: Events are grouped into space-time clusters representing coherent motions, with approximately constant motion parameters over a short processing window.Clusters have a tubular shape because events from the same moving edge trace trajectories through the space-time volume.
- Motion-compensation objective: Motion compensation separates multiple objects by warping events to a reference time and maximizing their alignment into sharp cluster-specific IWEs.A candidate warp matching an object's motion produces aligned events and a sharp IWE; incorrect motion produces blur.
- Alternating optimization: The method jointly estimates event-cluster associations and cluster motion parameters by maximizing the sum of cluster image contrasts.The optimization has no closed-form solution, so the method alternates between association refinement and motion-parameter updates.
- Alternating optimization: Each iteration computes soft event assignments from local cluster sharpness, then updates all cluster motion parameters in an ascent direction.The association update resembles an E-step and the motion update an M-step in an EM-like procedure.
- Limitations: The alternating method converges only locally, requires initialization, and uses the number of clusters as a hyper-parameter manually set in the experiments.The method is reported to be insensitive to excess clusters, but automatically selecting the optimal count is outside the paper's scope.
- Design choices: The method classifies events by motion without explicitly computing optical flow, using cluster warps instead of an error-prone optical-flow estimation step.This distinguishes the approach from simply clustering optical flow.
- Outputs: Motion-compensated IWEs recover fine appearance details of event-generating objects without estimating a computationally expensive 3D scene representation.The recovered patterns can describe object intensity structure such as shape, and are not limited to only the dominant motion cluster.
4. Experiments
Experiments evaluate the method on challenging event-camera data, measuring benchmark performance, per-event accuracy, computational throughput, and robustness across scenes, motion models, and cluster counts.
- Real-world sequences: The method segments independently moving objects and camera-induced background motion in difficult scenes, including occlusions, strobe lighting, traffic, HDR, and high-speed motion.Examples include separating a ball from a net, a quadrotor from camera motion, vehicles traveling at different speeds, and objects crossing in front of the sun.
- Per-event accuracy: 90% segmentation accuracy requires roughly 4 pixels of relative displacement, consistently across the tested relative velocities.The evaluation counts correctly classified events directly rather than using bounding-box accuracy.
- Computational performance: Processing reaches 240,000 events/s on a single 2.4 GHz CPU core for optical-flow-type warps, with complexity O((N_e + N_p)N_ℓN_it).Throughput decreases as the number of clusters increases because events are warped for every cluster and optimization dimensionality grows.
- Real-world sequences: Different clusters can use different motion models, including rotational motion for a fan and linear velocity for a falling coin.The method converges to the expected solution while handling occlusions on the fan blades.
- Cluster sensitivity: The method is robust to cluster-count choices: too few clusters discretize a motion continuum, while excess clusters collapse without assigned events.This behavior supports using a large cluster count, although depth estimation is better handled by tailored methods.
- Motion-model sensitivity: Using only rotation motion models fails on the tested sequence, whereas mixing optical-flow and rotation models leaves unsuitable rotation clusters unused.The authors identify automatic selection of relevant motion models as future work.
5. Conclusion
The method jointly segments events and recovers object motions, while producing sharp motion-compensated images. It outperforms recent methods by as much as 10% and resolves small relative motion differences.
- The method jointly segments events and recovers the motion parameters of the objects or clusters causing them.
- As a by-product, it produces motion-compensated images with sharp edge-like object appearances that may support further analysis such as recognition.
- As much as 10% improvement over the state-of-the-art was achieved on a publicly available dataset.
- The method resolves small relative motion differences between clusters and avoids explicit optical-flow estimation for motion segmentation.
B. Two Additional Motion-Compensation Segmentation Methods
Two classical clustering methods, mixture densities and fuzzy k-means, are adapted for event-based motion segmentation by leveraging motion compensation.
- Mixture densities and fuzzy k-means are modified to address event-based motion segmentation.
- Both adaptations leverage motion compensation to segment events.
B.1. Mixture Densities
The mixture-density method models events as a mixture of motion-coherent clusters and uses motion-compensated event alignment to estimate cluster memberships and motion parameters iteratively.
- The mixture density models the event distribution as a sum of cluster-specific densities with mixing weights and independent cluster parameters.
- Events are represented in the image-plane space-time volume, rather than as optical-flow vectors, and each cluster describes a coherent motion.
- The unweighted IWE measures how well a cluster explains an event through alignment at the event’s warped location.
- Each mixture component is constant along the point trajectories defined by its cluster warping model, producing the intended tubular structure.
- The EM procedure iteratively updates mixing weights and motion parameters until convergence, while also estimating the component distributions.
- During EM iterations, component distributions become sharper around segmented objects and blurred around non-segmented objects.
- The fuzzy k-means alternative maximizes an alignment-based objective and iteratively updates event assignments and cluster motion parameters.
- The logarithm of the IWE reduces the influence of large values counted repeatedly when events warp to the same pixel.
B.3. Comparison of Three Motion-Compensation Segmentation Methods
The proposed layered method is compared with mixture-density and fuzzy k-means alternatives. On a toy example, its event-cluster associations split objects better, while all methods converge after about 20 iterations and the proposed method reaches the highest objective score.
- The three motion-compensation segmentation methods are compared on a toy scene containing three objects moving in different image-plane directions.
- The proposed method yields the best object separation because event-cluster associations are included in each cluster’s motion-compensated image.The alternatives show all objects in every IWE, whereas the proposed method produces only minor ghost effects.
- The three per-event segmentation methods are presented as novel, with the main paper focusing on Algorithm 1 and supplementary material covering the classical adaptations.
- The core computation for all three methods is the generation of one warped-event image per cluster, with complexity O(NeNℓ).
- After approximately 20 iterations, all methods stagnate on the traffic sequence, while the proposed method achieves the highest objective score.
C.1. Non-rigid Moving Objects
The method applies low-dimensional motion warps to segment non-rigid scenes, working well when motion is approximately rigid over short event windows but showing uncertainty or failure during deformation.
- Model scope: Simple parametric warps are robust for rigid motion but have limited expressiveness for deformations such as pedestrians, birds, and jellyfish.More expressive models could represent complex motion, but they increase search-space dimensionality and local-minimum risks.
- Pedestrian: Pedestrian walking was separated into background, torso, and swinging-arm clusters because motion is approximately rigid over milliseconds.The camera was panning during the sequence.
- Popping balloon: Balloon-popping segmentation struggles during puncture but reasonably separates fast-moving rubber fragments afterward.The example tests the limits of the approximate-rigidity assumption.
- Depth variation: Continuous depth variation is discretized into depth slices whose event likelihood fades between foreground and background clusters.The fading reflects lower association certainty for events between clusters.
- High-resolution data: The method also segments 640 × 480 event data from a Samsung DVS Gen3 while the camera moves through scenes with continuous depth gradients.The high-resolution examples include a textured carpet, leaves, and a temple poster.
C.4. Comparison to k-means Optic Flow Clustering
The experiments compare direct motion-based event segmentation with clustering of separately estimated optical flow. Direct segmentation resolves objects with similar apparent velocities more accurately, while flow clustering can succeed when velocities are sufficiently distinct.
- Method comparison: Figure 10 visualizes motion-corrected events across iterations for three segmentation algorithms, enabling comparison of convergence behavior.Rows correspond to Algorithms 1–3, and columns show iterations 1, 5, 10, 20, and 80.
- Numbers sequence: Six numbers at different depths produce six expected apparent velocities because parallax makes closer objects appear faster.The camera moved back and forth at approximately constant speed on a linear slider.
- Numbers sequence: The two-step optical-flow-plus-k-means approach merges numbers 3, 4, and 5 and overrepresents the fastest-moving zero.The corresponding velocities are too similar to resolve correctly.
- Numbers sequence: Algorithm 1 accurately clusters the numbers by object velocity and produces sharper motion-compensated event images than the two-step approach.The comparison uses an optical-flow warp with linear image-plane motion.
- Rocks at different speeds: K-means can resolve objects when their relative velocities are sufficiently distinct, but struggles when similar velocities make flow clusters overlap.The rocks sequences use relative velocities of 50 pixels/s and 6 pixels/s.
- Additional evaluations: The broader examples include non-rigid scenes, continuous depth, high-resolution data, and optical-flow clustering comparisons.These figures cover walking and balloon popping, checkerboard translation, Samsung DVS data, numbers, and rocks.