Source-linked AI summary
EV-SegNet: Semantic Segmentation for Event-based Cameras
Iñigo Alonso, Ana C. Murillo
TL;DR
Semantic segmentation with event cameras lacks established representations and labeled data. The paper addresses these gaps with an event-only CNN, a new representation, and approximate DDD17 labels; it reports effective learning, improved event representations, and complementary event and grayscale results.
Problem
Semantic segmentation on event cameras lacks a clearly adopted CNN-compatible event representation and labeled training data.
Method
The paper builds an Xception-based encoder-decoder using event data, proposes a six-channel representation, and generates approximate labels for DDD17 sequences.
Results
The approach enables effective semantic-segmentation learning from event data, and the proposed representation outperforms previously used representations while event and grayscale inputs show complementary benefits.
Takeaways & Limitations
Event data can support semantic segmentation on its own, while combining event and conventional-image information is beneficial for the task.
Takeaways & Limitations
Training and evaluation use sequences that can be properly segmented in grayscale, excluding nighttime images because ground truth is unavailable.
Abstract
from arXiv · showhide
Event cameras, or Dynamic Vision Sensor (DVS), are very promising sensors which have shown several advantages over frame based cameras. However, most recent work on real applications of these cameras is focused on 3D reconstruction and 6-DOF camera tracking. Deep learning based approaches, which are leading the state-of-the-art in visual recognition tasks, could potentially take advantage of the benefits of DVS, but some adaptations are needed still needed in order to effectively work on these cameras. This work introduces a first baseline for semantic segmentation with this kind of data. We build a semantic segmentation CNN based on state-of-the-art techniques which takes event information as the only input. Besides, we propose a novel representation for DVS data that outperforms previously used event representations for related tasks. Since there is no existing labeled dataset for this task, we propose how to automatically generate approximated semantic segmentation labels for some sequences of the DDD17 dataset, which we publish together with the model, and demonstrate they are valid to train a model for DVS data only. We compare our results on semantic segmentation from DVS data with results using corresponding grayscale images, demonstrating how they are complementary and worth combining.
1. Introduction
The paper introduces semantic segmentation from event-camera data, addressing representation and labeling challenges with a CNN, a new event representation, and automatically generated approximate labels.
- Event cameras capture asynchronous intensity changes with high temporal resolution, difficult-lighting capability, and low power and bandwidth requirements.They record streams of events rather than fixed-framerate images.
- Deep-learning recognition on event cameras is limited by the lack of a standard event representation for CNNs and labeled training data.The paper targets semantic segmentation, an application not previously explored with this sensor.
- The work builds an Xception-based CNN that uses DVS data for semantic segmentation and proposes automatically generated approximate labels from DDD17 sequences.The model and generated data are released.
- A new DVS representation is reported to outperform previously used representations, while comparisons analyze event data against conventional images and their combination.The study examines benefits and drawbacks of the modalities rather than treating event data as a standalone replacement.
2. Related work
Prior event-camera research largely addresses reconstruction, tracking, optical flow, detection, and recognition, while semantic segmentation has focused on RGB and multimodal imagery. This paper adapts an Xception-based encoder-decoder to event data for this previously unexplored segmentation setting.
- Event-camera applications: Recent event-camera applications mainly target 3D reconstruction and 6-DOF camera tracking, with additional work on optical flow, detection, and recognition.Recognition datasets remain scarce, though DDD17 provides more challenging real-world recordings.
- Event representations: Event-camera recognition methods commonly encode asynchronous events into dense, image-like representations containing spatial and sometimes temporal information.This encoding facilitates processing by conventional visual-recognition models.
- Semantic segmentation: Semantic segmentation assigns a semantic label to every image pixel, and state-of-the-art approaches commonly use encoder-decoder CNN architectures.The paper distinguishes class-level segmentation from instance-level approaches such as Mask-RCNN.
- Position of this work: Prior segmentation work mainly uses RGB images or adds modalities such as depth, whereas this paper applies an Xception-based encoder-decoder to event-camera data.The paper reports good segmentation using only event data from a public benchmark.
3. From Events to Semantic Segmentation
Event cameras output asynchronous intensity-change events rather than conventional images, so semantic segmentation requires dense representations and CNN adaptations. The paper combines spatial event histograms with timestamp statistics in a six-channel representation and processes them with an Xception-based encoder-decoder network.
- 3.1. Event Representation: Each event records pixel coordinates, polarity, and timestamp, but event streams are not directly suited to CNNs because a simple event matrix omits spatial relationships.Events encode changes in intensity rather than absolute intensity values.
- 3.1. Event Representation: Dense encodings aggregate events within a time window, with Hist storing positive and negative event occurrences in separate channels.Earlier one-channel encodings retain only the last event at each pixel.
- 3.1. Event Representation: The proposed representation combines two polarity histograms with timestamp information for all events, rather than retaining only one timestamp per pixel.This design is motivated by the usefulness of temporal information for recognition tasks.
- 3.1. Event Representation: The resulting event representation is a six-channel image containing positive and negative histograms plus per-polarity mean and standard deviation of normalized timestamps.The six channels are Hist(x, y, −1), Hist(x, y, +1), M(x, y, −1), M(x, y, +1), S(x, y, −1), and S(x, y, +1).
- 3.2. Semantic Segmentation from Event Data: Semantic segmentation uses an Xception-based encoder with a light decoder, skip connections, and an auxiliary loss to produce per-pixel predictions.Training uses soft-max cross entropy summed over per-pixel losses.
4. Ev-Seg: Event-Segmentation Data
Ev-Seg extends DDD17 with automatically generated semantic segmentation labels, grayscale images, and event data for training and evaluating event-based segmentation. The labels are produced from grayscale Cityscapes models under selected urban daytime conditions, with known domain-shift and annotation limitations.
- Dataset construction: Ev-Seg adds automatically generated semantic segmentation labels to a large subset of DDD17, alongside grayscale images and three event-data representations.The released extension is intended to support replication and further experimentation.
- Label generation: The labels are generated by training a grayscale Cityscapes segmentation CNN, applying it to selected DDD17 grayscale images, and using its predictions as event-model training labels.The pipeline also crops the bottom 60 image rows because they contain the car dashboard and add noise.
- Sequence selection: The DDD17 subset contains six urban daytime sequences selected for suitable illumination and contrast; five are used for training and one for testing.The test sequence was chosen for its more homogeneous class distribution, including more examples of less frequent human or pedestrian labels.
- Label scope and limitations: Ev-Seg labels use Cityscapes categories except that sky and construction are learned together because domain shift prevents reliable grayscale inference for those categories on DDD17.This domain shift also motivated generating Cityscapes categories rather than its original classes.
- Label quality: The generated labels are reasonably accurate and well defined, but less perfect than manual annotations, especially for building and sky classes.Figure 4 illustrates grayscale inputs and their corresponding generated segmentations for the test sequence.
5. Experimental Validation
Experiments evaluate event-camera semantic segmentation across metrics, representations, integration intervals, and conventional grayscale inputs. Event and grayscale cameras show complementary strengths, while combined sensing improves segmentation.
- Evaluation setup: Accuracy and MIoU are used to evaluate per-pixel semantic segmentation from event-camera data.MIoU is computed per class, alongside standard classification accuracy.
- Input representation comparison: The proposed event encoding performs slightly but consistently better than other event representations across semantic-segmentation metrics.Table 2 compares dense event-location, temporally enriched, and proposed encodings.
- Integration intervals: Models trained with 50ms intervals perform only slightly worse when tested with 10ms or 250ms intervals.Normalization and varied camera speeds in training are offered as explanations for generalization across intervals.
- Event versus conventional cameras: Event information helps segment moving pedestrians and refine object borders compared with grayscale inputs.Conventional cameras provide richer appearance information, whereas event cameras provide useful motion information.
- Complementary sensing: Event cameras struggle with stationary objects, while conventional cameras struggle under extreme illumination; their combination yields better segmentation.The evaluation set lacks night-time ground truth and therefore slightly favors grayscale inputs.
6. Conclusions and Future Work
The work establishes event-camera semantic segmentation using an Xception-based encoder-decoder, approximate DDD17 labels, and a representation combining event counts with temporal distribution. It reports stronger event representations and complementary benefits from combining event and conventional imagery, while identifying label generation as a major remaining challenge.
- Conclusions: The paper presents initial semantic-segmentation results using event-camera information and releases models with generated labeled data.Approximate labels are generated for selected DDD17 sequences because no benchmark exists.
- Conclusions: The proposed representation encodes both event histograms and temporal distributions, outperforming previously used event representations across different time intervals.The representation is evaluated for semantic segmentation using event data.
- Conclusions: Comparisons with conventional images show distinct benefits and drawbacks for event-based and conventional sensing, including benefits from combining both sensors.The paper illustrates adversarial cases involving static scenes and extreme lighting.
- Future work: Obtaining more and better semantic-segmentation labels remains a main challenge, motivating domain adaptation and event-camera simulators.Current event-camera simulators do not provide this type of label.