Source-linked AI summary
ESS: Learning Event-based Semantic Segmentation from Still Images
Zhaoning Sun, Nico Messikommer, Daniel Gehrig, Davide Scaramuzza
TL;DR
High-quality labeled datasets for event-based semantic segmentation are scarce, limiting work on a task relevant to challenging lighting and high-speed conditions. ESS transfers segmentation from labeled images to unlabeled events by aligning recurrent, motion-invariant event embeddings with image embeddings, and the paper introduces DSEC-Semantic. ESS outperforms existing UDA and supervised methods across the reported benchmarks.
Problem
Event-based semantic segmentation remains limited by a lack of high-quality labeled datasets, despite the promise of event cameras in challenging lighting and high-speed settings.
Source: §1 · p. 2Method
ESS transfers segmentation from labeled images to unlabeled events by aligning recurrent, motion-invariant event embeddings with image embeddings, without requiring video, paired data, or hallucinated motion.
Source: §1 · pp. 2–3Results
ESS outperforms existing UDA and supervised methods across DDD17 and DSEC-Semantic; on DDD17 it reports 6.98% higher mIoU than other UDA methods.
Source: §1 · p. 3, §5 · p. 14Takeaways & Limitations
DSEC-Semantic adds a large-scale, fine-grained event dataset, while ESS makes existing labeled image datasets available for event-camera research.
Source: §1 · p. 3
Abstract
from arXiv · showhide
Retrieving accurate semantic information in challenging high dynamic range (HDR) and high-speed conditions remains an open challenge for image-based algorithms due to severe image degradations. Event cameras promise to address these challenges since they feature a much higher dynamic range and are resilient to motion blur. Nonetheless, semantic segmentation with event cameras is still in its infancy which is chiefly due to the lack of high-quality, labeled datasets. In this work, we introduce ESS (Event-based Semantic Segmentation), which tackles this problem by directly transferring the semantic segmentation task from existing labeled image datasets to unlabeled events via unsupervised domain adaptation (UDA). Compared to existing UDA methods, our approach aligns recurrent, motion-invariant event embeddings with image embeddings. For this reason, our method neither requires video data nor per-pixel alignment between images and events and, crucially, does not need to hallucinate motion from still images. Additionally, we introduce DSEC-Semantic, the first large-scale event-based dataset with fine-grained labels. We show that using image labels alone, ESS outperforms existing UDA approaches, and when combined with event labels, it even outperforms state-of-the-art supervised approaches on both DDD17 and DSEC-Semantic. Finally, ESS is general-purpose, which unlocks the vast amount of existing labeled image datasets and paves the way for new and exciting research directions in new fields previously inaccessible for event cameras.
1 Introduction
Event-based semantic segmentation lacks high-quality labeled datasets, despite event cameras’ promise in challenging lighting and high-speed settings. ESS transfers segmentation from labeled images to unlabeled events, and the paper introduces DSEC-Semantic to address the data gap.
1 Introduction
- Existing event segmentation datasets are synthetic or use pseudo-labels from low-quality grayscale images, and methods trained on them typically show suboptimal performance.The passage identifies this data limitation as a barrier to progress in event-based semantic segmentation.Source: §1 · p. 2
- DSEC-Semantic is a new event-based segmentation dataset based on DSEC, with labels generated from high-quality RGB images and filtered by manual inspection.The paper introduces the dataset alongside ESS to address the shortage of high-quality event labels.Source: §1 · p. 2
- ESS aligns recurrent, motion-invariant event embeddings with image embeddings to transfer segmentation from labeled images without video, paired data, or hallucinated motion.The paper notes that generating events from single images is ill-posed and has been studied using adversarial learning, which is prone to mode collapse.Source: §1 · p. 3
- ESS reports 6.98% higher mIoU than other UDA methods on DDD17, plus 2.57% over supervised methods with event labels; on DSEC-Semantic, margins are 4.17% and 1.53%.The 4.17% comparison is against other UDA approaches; the 1.53% comparison is against state-of-the-art supervised methods.Source: §1 · p. 3
- ESS is presented as general-purpose, making existing labeled image datasets available for event-camera research and supporting research in fields previously inaccessible to event cameras.The paper frames this as a consequence of using labeled image data rather than requiring event labels.Source: §1 · p. 3
2 Related Work
Prior event-segmentation work is constrained by low-quality or synthetic labels, paired-data requirements, video availability, or the need to hallucinate motion. ESS instead transfers from still images to unlabeled events using recurrent motion-invariant embeddings.
2.1 Event-based Semantic Segmentation
- Earlier event-segmentation datasets include pseudo-labels from low-resolution grayscale frames that produce artifacts and merge classes, while EventScape’s labels come from simulation and face a sim-to-real gap.The first event-segmentation work introduced DDD17; EventScape provides high-quality labels but was recorded in CARLA.Source: §2.1 · pp. 3–4
- Prior segmentation approaches either augment training with synthetic events from labeled video, require per-pixel paired events and images, or use labeled event data without recurrent event embeddings.The passage contrasts these requirements with ESS’s use of still-image datasets, unpaired events and images, and recurrent motion-invariant embeddings.Source: §2.1 · p. 4
2.2 Unsupervised Domain Adaptation
- Unsupervised domain adaptation transfers from a labeled image source domain to an unlabeled event target domain.E2VID reconstructs video from events but does not use unlabeled target events for task transfer; VID2E uses labeled video to generate synthetic events and cannot adapt to an unlabeled event domain.Source: §2.2 · p. 4
- Network grafting requires paired events and images, while EvDistill reports segmentation performance using paired data and thus is not strictly UDA.The paired-data requirement limits the datasets network grafting can use to those with per-pixel-aligned events and frames.Source: §2.2 · pp. 4–5
- ESS avoids generating fake events from still images by aligning recurrent, motion-invariant event embeddings with image embeddings.The related-work discussion describes motion hallucination as ill-posed and as hindering feature alignment.Source: §2.2 · p. 5
3 Approach
ESS trains an event-segmentation model using labeled source images and unlabeled target events. It uses a recurrent E2VID encoder to produce motion-invariant event embeddings, then aligns them with image embeddings using consistency losses and image-label supervision.
3 Approach
- ESS transfers pixel-wise semantic prediction from labeled source images to unlabeled event sequences recorded by an event camera.The source domain contains images and semantic maps; event streams comprise temporally ordered events encoding pixel coordinates, timestamps, and brightness-change polarity.Source: §3 · pp. 5–6
3.1 Network Overview
- The E2VID encoder processes successive event-window voxel grids recurrently to produce multi-scale event embeddings, which the task network uses for predictions.An E2VID decoder reconstructs an image from the event embedding; an image encoder re-encodes that reconstruction, and a shared task network predicts from both embeddings.Source: §3.1 · pp. 6–7
3.2 Aligning Motion-Invariant Embedding
- ESS aligns event and re-encoded-image embeddings and their task predictions with consistency losses, and also matches intermediate task-network features.The losses include an L1 embedding distance, a symmetric Jensen-Shannon divergence between predictions, and L1 matching of intermediate features.Source: §3.2 · p. 7, §3.3 · p. 7
3.3 Losses and Optimization
- Training combines image-label task loss with embedding, prediction, and feature consistency losses through two-stage gradient accumulation.The first stage uses labeled images to update task-network gradients; the second uses unlabeled events, freezes the E2VID encoder/decoder pair and one task-network branch, and accumulates gradients for the image encoder and task decoder.Source: §3.3 · p. 7
4 Experiments
The experiments evaluate ESS on DDD17 and DSEC-Semantic in unsupervised domain adaptation and supervised settings, and test its design through ablations. ESS outperforms competing methods in the reported comparisons, while label quality affects how some DDD17 predictions are scored.
4.1 DDD17 for Semantic Segmentation
- 6.98% mIoU over VID2E on DDD17 UDA, 4.17% over UDA methods on DSEC-Semantic, and 1.53% over EV-SegNet on DSEC-Semantic with image and event labels.The DDD17 UDA results use Cityscapes labels and unlabeled DDD17 events; DSEC-Semantic UDA uses image labels and unpaired, unlabeled events.Source: Table 1 · p. 9, §4.1 · p. 9, Table 3 · p. 12
- 2.57% mIoU over DTL on DDD17 when ESS is trained with labeled events.The experiment also tests adding image labels, but the authors report no performance improvement and link this to inaccuracies in DDD17 semantic labels.Source: Table 2 · p. 9, §4.1 · p. 10
- DDD17’s low-quality labels sometimes omit visible objects, so finer event-and-image predictions can receive lower detection scores despite showing those objects.The paper notes that the predictions can be more accurate than the DDD17 pseudo-labels in these examples.Source: §4.1 · pp. 10–11, Figure 4 · p. 10
4.2 DSEC-Semantic
- DSEC-Semantic uses RGB-derived labels for 11 classes and includes 8,082 training and 2,809 test frames from daytime sequences.The labels are generated from warped high-resolution color images using a frame-based segmentation method; the dataset was introduced because DDD17 labels have low-quality and low-resolution artifacts.Source: §4.2 · p. 11
4.3 Ablation Studies
- 5.56%, 1.28%, and 2.09% mIoU drops follow removal of the embedding, prediction, and task consistency losses, respectively, in DSEC-Semantic UDA.The ablation reports these losses as important to the framework.Source: §4.3 · p. 13, Table 5 · p. 14
- 0.025 symmetric KL divergence for ESS versus 0.073 for E2VID indicates three times lower disagreement between image- and event-based predictions on DSEC-Semantic.The comparison measures consistency across paired predictions decoded from image and event embeddings.Source: Table 6 · p. 14, §4.3 · p. 14
5 Conclusion
ESS transfers semantic segmentation from labeled images to events, and the paper evaluates it on DDD17 and DSEC-Semantic. The authors report gains over state-of-the-art UDA and supervised methods and present DSEC-Semantic as a high-quality labeled dataset.
5 Conclusion
- ESS outperforms state-of-the-art UDA and supervised methods on DDD17 and DSEC-Semantic.The paper introduces DSEC-Semantic with high-quality, fine-grained semantic labels.Source: §5 · p. 14
- The method transfers labeled image datasets to event-based semantic segmentation without event labels or paired events and images.The authors say it unlocks existing image datasets for event-based vision and could support exploration of other tasks, though only semantic segmentation is evaluated.Source: §5 · p. 14
- DSEC-Semantic is a large-scale event-based dataset with high-quality, fine-grained semantic labels.The authors state that it will spur further research in event-based semantic scene understanding.Source: §5 · p. 14
6 DSEC-Semantic
DSEC-Semantic is built from DSEC sequences by projecting frame-camera images into the event-camera view and generating semantic labels. The dataset uses day sequences and converts 19 fine-grained labels into 11 classes for experiments.
6 DSEC-Semantic
- DSEC-Semantic labels are generated by warping 1440 × 1080 left-camera images to the 640 × 480 event-camera view, then cropping the last 40 rows.The resulting labels have a resolution of 640 × 440.Source: §6 · p. 15
- A state-of-the-art semantic segmentation method generates labels from the warped images, initially producing 19 classes.The 19 classes match Cityscapes labels for evaluation; experiments convert them to 11 classes: background, building, fence, person, pole, road, sidewalk, vegetation, car, wall, and traffic sign.Source: §6 · p. 15
- To preserve label quality, the dataset includes only DSEC sequences recorded during the day.It contains 8,082 labeled training frames and 2,809 labeled test frames.Source: §6 · p. 15
7 Event Representation
The method converts event streams into sequences of grid-like representations, including voxel grids, using non-overlapping windows with a fixed number of events. Its representation is described using temporal bins over an event time window.
7 Event Representation
- An event stream E is converted into a sequence of grid-like representations, including voxel grids Vk.The grids are constructed from non-overlapping event windows containing a fixed number of events.Source: §7 · p. 15
- Each voxel grid is built from a non-overlapping window containing a fixed number of events.Source: §7 · p. 15
- The representation uses B bins over an event time window ∆T, with t0 denoting the first event's time in that window.Source: §7 · p. 15
8 Network Architecture
The network is fully convolutional and U-Net-inspired, combining an E2VID event encoder-decoder with an image encoder and task network. The event encoder provides recurrent multi-scale embeddings, while ResNet-18 features provide image embeddings and skip connections.
8 Network Architecture
- The network is a fully convolutional architecture inspired by U-Net.Source: §8 · p. 15
- The E2VID encoder supplies recurrent, multi-scale event embeddings from three recurrent encoder layers.The E2VID decoder comprises two residual blocks, three decoder layers, and a final image-prediction layer.Source: §8 · pp. 15–16
- The image encoder uses ResNet-18 features through its sixth residual block, with outputs from the second and fourth residual blocks used as task-network skip connections.The task network consists of five residual blocks, seven convolution layers, and three upsampling layers.Source: §8 · p. 16
9 Training Details
Training details specify separate source–target setups and optimization configurations for DDD17 and DSEC-Semantic.
9 Training Details
- DDD17 uses Cityscapes as the labeled source and DDD17 as the unlabeled target, representing each sample with 20 voxel grids of 32,000 events.Source: §9 · p. 17
- DDD17 training uses RAdam for 50,000 iterations with batch size 16; learning rates are 1×10−5 for Eimg and 1×10−4 for T.The authors report that a smaller Eimg learning rate and first-stage gradient accumulation for Eimg help improve results.Source: §9 · p. 17
- For UDA comparisons on DDD17, the E2VID baseline uses a U-Net trained on grayscale Cityscapes images and labels, while uses the same source and target domains.Source: §9 · p. 17
- DSEC-Semantic training uses Cityscapes as source and DSEC-Semantic as target, with 100,000 events per voxel grid and every second selected-sequence sample skipped.The first six samples of each sequence are removed; the resulting sets contain 4,017 training and 1,395 test samples.Source: §9 · p. 17
- DSEC-Semantic training uses RAdam for 25,000 iterations, batch size 8, and a learning rate of 5×10−4 for both Eimg and T.The four λ hyperparameters are each set to 1.Source: §9 · p. 17
10 E2VID Driving Dataset
The E2VID driving dataset contains unpaired, unlabeled events and images from separately recorded cameras, yet ESS transfers the Cityscapes-trained task to its events.
10 E2VID Driving Dataset
- The E2VID driving sequences contain events from a Samsung DVS Gen3 and images from a Huawei P20, with neither external calibration nor time synchronization.Source: §10 · p. 17
- Despite the cameras being uncalibrated and unsynchronized, ESS transfers the task trained on Cityscapes images to the E2VID event data.Source: §10 · p. 17
- The dataset provides a test of transfer using completely unpaired and unlabeled events and images.Source: §10 · p. 17