PaperlayerView exact revision on arXiv

Source-linked AI summary

Event-SAMA: Language-Grounded Video Object Segmentation on Neuromorphic Event Streams

Sergei Fedorchenko, Max Zehnder, Nikola Zubić, Rong Zou, Davide Scaramuzza

arXiv:2610.39676v1cs.LGcs.AIcs.CLcs.CV

TL;DR

Referring video object segmentation lacks datasets and models for event streams, despite event cameras' microsecond temporal resolution and high dynamic range. This paper converts annotated SAMA training data into synthetic events and adapts SAMA-4B to segment from event inputs. Event-SAMA reaches 24.37 J&F on MeViS valid_u and 22.33 J&F on the converted ReVOS valid subset.

  • Problem

    No dataset or model addresses natural-language referring segmentation on event streams, although event cameras offer microsecond latency, high dynamic range, and no motion blur.

    Source: §1 · p. 2

  • Method

    The paper converts the SAMA corpus and additional training data into synthetic event streams and adapts SAMA-4B with event-specific input representations.

    Source: §1 · p. 2

  • Results

    24.37 J&F on MeViS valid_u and 22.33 J&F on the converted ReVOS valid subset are reported for full-corpus Event-SAMA.

    Source: §5.2 · p. 6

  • Takeaways & Limitations

    Event-SAMA establishes a first event-native RVOS baseline, with evaluation on real event recordings working without retraining when the accumulation window matches event density.

    Source: §6 · p. 8, §7 · p. 9

    All training and test events are synthetic, and the small real-event evaluation covers 25 MouseSIS clips from one animal domain and 200 N-Caltech101 samples.

    Source: §5.2 · p. 6, §6 · p. 8

Abstract

from arXiv · show

Event cameras offer microsecond temporal resolution, high dynamic range and no motion blur, but referring video object segmentation (RVOS) lacks event-stream datasets and models. We introduce Event-SAMA-239K, a synthetic-event conversion of the SAMA-239K grounded video-chat corpus and all additional SAMA training data: 30,650 video clips and 392,162 images. Optical-flow, FILM or saccade upsampling followed by a linear event generator preserves all mask, referring-expression and dialogue annotations. We also introduce Event-SAMA, adapting SAMA-4B with a two-channel polarity patch embedding initialised from pretrained RGB weights and a zero-initialised event adapter. End-to-end training yields 24.37 $\mathcal{J}\&\mathcal{F}$ on MeViS valid_u and 22.33 $\mathcal{J}\&\mathcal{F}$ on the converted ReVOS valid subset, establishing the first event-native RVOS baseline (RGB SAMA-4B: 55.4 on MeViS). Frozen RGB SAMA-4B with pseudo-RGB renderings or E2VID reconstructions of the same events scores 13.23 and 15.94 $\mathcal{J}\&\mathcal{F}$. Without retraining, Event-SAMA works on real event-camera recordings when the accumulation window matches their event density. A seed-controlled ablation with a measured noise floor of 0.47 $\mathcal{J}\&\mathcal{F}$ finds gains from diverse training sources over MeViS alone (+4.99), RGB-mean event-stem initialisation (+4.33), and a trainable SAM2 mask decoder (+2.76); the event adapter, partial ViT unfreezing and event polarity are within noise at this budget. An earlier, partially converted corpus snapshot trained with the identical recipe scores 1.38 $\mathcal{J}\&\mathcal{F}$ higher on MeViS and 1.12 higher on ReVOS. Both are single runs that also differ in schedule length, so we draw no conclusion about corpus scale.

1 Introduction

RGB-based referring segmentation models assume dense synchronous frames, while event streams offer different sensing characteristics and lack language-grounded datasets and models. The paper converts annotated SAMA data and adapts SAMA-4B to event inputs, alongside an event-specific evaluation protocol and seed-controlled ablations.

1 Introduction

  • Event cameras provide asynchronous, microsecond-latency brightness changes, but no dataset or model addressed natural-language referring segmentation on event streams.The paper motivates conversion of richly annotated RGB data as an alternative to recapturing and re-annotating video with event hardware.Source: §1 · p. 2
  • 30,650 video clips and 392,162 images from the SAMA-239K corpus and full SAMA training mix are converted to synthetic events while preserving mask, referring-expression, and dialogue annotations.Conversion uses optical-flow, FILM, or saccade upsampling followed by a linear event generator.Source: §1 · p. 2
  • Event-SAMA adapts SAMA-4B with a two-channel ON/OFF polarity stem initialized from pretrained RGB weights and a zero-initialized adapter for event features.The adapter aligns event features with the frozen vision-language pipeline.Source: §1 · p. 2
  • The paper introduces seed-controlled ablations and an event-specific protocol to distinguish run-to-run noise and assess sensitivity to keyframe count and accumulation-window width..Source: §1 · p. 2

2 Related work

Prior work develops language-grounded segmentation and dialogue on RGB video, while event research focuses largely on other tasks. Existing real-event label-transfer approaches provide class labels, not referring expressions or dialogue about individual instances.

2 Related work

  • SAMA extends Sa2VA to multi-turn referential dialogue and contributes SAMA-239K, while Event-SAMA changes the input modality rather than the underlying architecture or source annotations.The paper builds on SAMA-4B, which uses a language model and promptable segmenter for mask prediction.Source: §2 · p. 2
  • EventGPT couples an LLM with event streams for text-only captioning, reasoning, and visual question answering, but not pixel-level segmentation or temporal object tracking.Prior event research has otherwise concentrated on optical flow, reconstruction, and detection.Source: §2 · p. 2
  • Real-event label-transfer pipelines provide class labels, but referring expressions and dialogue about individual instances cannot be derived from frames and require human annotation for each clip.The paper converts an existing referring corpus for training and uses frame-annotated MouseSIS data for real-event evaluation.Source: §2 · p. 3

3 Event-SAMA-239K

Event-SAMA-239K converts the annotated SAMA-239K core and the additional SAMA training mix into synthetic event streams while retaining the source annotations. Its conversion uses different upsampling strategies for video and still images, and validation finds both real-event density overlap and limitations of the simulations.

3.1 Sources and scope

  • SAMA-239K unifies five video datasets, while the full training mix also includes video and image instruction-tuning sources; both tiers are converted with their original annotations preserved.The annotated core is the basis for the dataset name; additional unannotated clips are converted but not counted in the reported training composition.Source: §3.1 · p. 3

3.2 Conversion pipeline

  • Video sources use DIS optical-flow upsampling to at least 200 fps, except ChatUniVi, which uses FILM interpolation; still-image sources use synthetic saccades before event generation.The linear event generator emits ON and OFF events when pixel brightness changes cross dataset-specific contrast thresholds.Source: §3.2 · pp. 3–4

3.3 Validation of the converted streams

  • Converted video-source medians range from 4.9 ev/px/s for ChatUniVi to 55.1 ev/px/s for Ref-YouTube-VOS, and ChatUniVi overlaps substantially with real Gen1 recordings at 4.9 versus 8.1 ev/px/s.Linear thresholding also fires on optical-flow warping residuals and JPEG artefacts, so equal event density does not mean equal scene-motion encoding.Source: §3.3 · p. 4
  • Image-source densities are 29.9–64.5 ev/px/s because the synthetic saccade shifts many pixels across the contrast threshold, making these streams denser than typical real recordings..Source: §3.3 · p. 4

4 Event-SAMA

Event-SAMA adapts SAMA-4B to predict masks from event frames and natural-language expressions, retaining its language and segmentation pipeline while replacing RGB input with event-specific representations. The architecture combines per-keyframe and clip-context features with a SAM2 grounding path.

4 Event-SAMA

  • Given an event stream and referring expression, Event-SAMA predicts a binary mask for the referred object at each annotated timestamp, without RGB frames at training or test time.The model adapts SAMA-4B to consume event frames in place of RGB images.Source: §4 · p. 4
  • Event frames feed a semantic path that produces keyframe and whole-clip features for a frozen language model, whose [SEG] token prompts the mask decoder.The frozen SAM2 image encoder supplies dense embeddings from pseudo-RGB event renderings, while the trained decoder produces per-frame masks.Source: Figure 1 · p. 5, §4 · p. 5
  • The two-channel polarity stem replaces the RGB patch embedding with ON/OFF event channels and initializes both channel kernels from the average pretrained RGB kernel.This initialization is reported to improve J&F by 4.33 over random initialization.Source: §4 · p. 5
  • A zero-initialized residual event adapter after mlp1 starts as the identity and can learn a modality-specific correction on keyframe features.Its ablation changes are within the reported seed noise, so the paper retains it on architectural rather than empirical grounds.Source: §4 · p. 5
  • Training freezes the LLM, InternViT body, and SAM2 image encoder while training 323M parameters across the event stem, projections, aggregator, adapter, and mask decoder.The pretrained [SEG] token embedding remains frozen; only its projection into SAM2 prompt space is trained.Source: §4 · p. 6

5 Experiments

Experiments evaluate Event-SAMA on standard referring-VOS benchmarks, real event-camera recordings, and controlled ablations. The full-corpus model is the main benchmark result, while real-data tests and ablations reveal practical conditions and factors that matter at the tested budget.

5.1 Setup

  • The evaluation uses MeViS valid_u and a 97-video converted subset of ReVOS valid; unconverted ReVOS videos are excluded rather than counted as misses.Masks are predicted on 32 uniformly spaced keyframes and assigned to remaining annotated timestamps by nearest-keyframe propagation.Source: §5.1 · p. 6

5.2 Main results

  • 24.37 J &F on MeViS valid_u and 22.33 J &F on the converted ReVOS subset are the full-corpus Event-SAMA results; pseudo-RGB and E2VID reuse score 13.23 and 15.94 J &F versus 24.37.Table 2 also reports a 55.4 J &F RGB SAMA-4B MeViS upper bound, which is not a like-for-like comparison; ReVOS scoring uses only converted videos.Source: §5.2 · pp. 6–7, Table 2 · p. 7

5.3 Real event-camera data

  • 31.7 J &F on 25 MouseSIS clips exceeds the 24.5 language-free motion-mask score, while N-Caltech101 correct-class expressions beat wrong-class expressions in 69% of samples.MouseSIS uses accumulation windows matched to each clip's sparser event density rather than the 10 ms training window.Source: §5.3 · p. 7

5.4 Ablations

  • The earlier corpus snapshot scored 1.38 J &F higher on MeViS and 1.12 higher on ReVOS, but differing snapshot, schedule length, and seed preclude a conclusion about training-set size.Each run was trained once; the comparison is interpreted as run-to-run variation against a measured 0.47 J &F two-seed spread.Source: §5.4 · p. 7
  • +4.33 J &F from RGB-mean stem initialisation and +2.76 J &F from training the SAM2 mask decoder both exceed the measured 0.47 J &F seed spread.The ablation uses the five-source subset and a 1-epoch iso-compute recipe; its baseline is distinct from the full-corpus headline model.Source: §5.4 · pp. 7–8, Table 3 · p. 8
  • At this budget, adapter removal (+0.67), last-six-layer ViT unfreezing (−0.71), and polarity merging (+0.52 J &F) are within the 0.47 J &F seed spread or underpowered.Adapter and ViT deltas reversed sign in an earlier batch, and a single polarity run cannot establish that polarity has no effect.Source: §5.4 · pp. 7–8, Table 3 · p. 8

6 Limitations

The evaluation is bounded by synthetic event generation, a small real-event test, and incomplete benchmark coverage. The ablation budget and remaining gap to RGB also limit what the results establish.

6 Limitations

  • All training and test events are synthetic, and the linear generator omits sensor noise, refractory periods, threshold mismatch, and real microsecond timing statistics.Matching event density to Gen1 recordings validates only one axis of similarity.Source: §6 · p. 8
  • Real-event evaluation covers only 25 MouseSIS clips from one animal domain and 200 N-Caltech101 samples; accumulation windows must match sensor event density.Training on real or density-randomised events is identified as a follow-up direction.Source: §6 · p. 8
  • Still-image sources form most of the corpus by count, and their synthetic saccades produce higher event densities than video sources.Image-heavy training conclusions should be read with this proxy in mind.Source: §6 · p. 8
  • ReVOS results cover 97 of 416 videos, and the trained grounded-dialogue capabilities have not been benchmarked.MeViS valid_u has full coverage at 793 items.Source: §6 · p. 9
  • Ablations use one epoch, two baseline seeds, and one seed per variant, so effects below approximately 0.5 J &F cannot be resolved.Three of six architectural ablations are reported as inconclusive.Source: §6 · p. 9
  • At 24.37 J &F versus the RGB upper bound of 55.4, Event-SAMA is a feasibility baseline rather than a competitive system.A large performance gap remains.Source: §6 · p. 9

7 Conclusion

The paper introduces a synthetic event corpus and an event-native language-grounded segmentation model, reporting results on MeViS and a converted ReVOS subset. It releases tools to regenerate the corpus rather than distributing the event streams themselves.

7 Conclusion

  • Event-SAMA-239K converts 30,650 video clips and 392,162 images while preserving SAMA-239K annotations, and Event-SAMA segments event streams from natural-language referring expressions.The reported scores are 24.37 J &F on MeViS valid_u and 22.33 J &F on the converted ReVOS valid subset.Source: §7 · p. 9
  • The event-native model outperforms frozen RGB SAMA-4B pipelines using pseudo-RGB renderings or E2VID reconstructions on MeViS, and works on real recordings without retraining when accumulation windows match event density.Ablations identify data diversity (+4.99), RGB-mean event-stem initialisation (+4.33), and a trainable SAM2 mask decoder (+2.76) as decisive; other tested factors are within a 0.47-J &F noise floor.Source: §7 · p. 9

A Licensing and release

  • The release provides the conversion pipeline, training code, evaluation harness, checkpoints, and a manifest for regenerating the corpus from original data obtained from its providers.The 17.41 TB event streams are not released because they are much larger than their RGB sources and some source datasets restrict redistribution of derived works.Source: A Licensing and release · p. 11

B Implementation details

The implementation combines event-derived visual features with SAMA’s spatio-temporal aggregation and language pathway. It also specifies training and evaluation settings, including a separate pseudo-RGB rendering for the grounding path.

B Implementation details

  • SAMA’s STC aggregator uses spatial and temporal Q-Formers to turn per-frame InternViT features into a fixed-size clip summary mapped into the language embedding space.The Q-Formers and projection are trained from scratch and operate on event-derived ViT features without modification.Source: B Implementation details · p. 11
  • The event adapter follows mlp1 in a sequential module, so it runs on every inference code path; mlp1 is trained jointly with the adapter.Source: B Implementation details · p. 11
  • The grounding path renders each event frame as pseudo-RGB for SAM2, while the semantic and language path consumes the two-channel event frame directly.SAM2 then applies its standard resizing and normalisation.Source: B Implementation details · p. 11
  • Full-corpus training runs for four epochs and 133,208 iterations; the five-source comparison also runs for four epochs, while ablations use one epoch.Training uses 16 GH200 GPUs across four nodes, bf16, global batch 48, and a cosine schedule with warm-up.Source: B Implementation details · p. 11
  • Evaluation predicts masks on 32 uniform keyframes accumulated over 10 ms, gives the LLM the first five as visual context, and assigns each annotated frame its nearest keyframe prediction.Scores use the standard MeViS evaluation code.Source: B Implementation details · p. 11

C Qualitative results

The qualitative example contrasts the RGB reference with the sparse event input, where motion outlines the referred object against a nearly empty static background. Protocol sweeps show that accuracy responds asymmetrically to accumulation-window width and increases with keyframe count.

C Qualitative results

  • The event frame shows motion-induced events along the moving elephant’s contours while the static background is almost empty; the RGB frame is reference-only and is not model input.The model must ground the expression in shape and motion rather than colour or texture.Source: C Qualitative results · p. 11, Figure 2 · p. 12

D Protocol sensitivity

  • 23.04 J &F at K = 32 exceeds 20.64 at K = 8 and 21.93 at K = 16; doubling the accumulation window to 20 ms gains 1.55 J &F, while halving it to 5 ms costs 4.31 J &F.K = 64 could not be evaluated because the SAM2 image encoder did not run at that batch size in this setup.Source: D Protocol sensitivity · p. 12, Table 4 · p. 12

E Real event-camera evaluation: details

The real-event evaluation uses MouseSIS recordings and N-Caltech101 sensor-motion recordings, with per-frame masks and language queries. Results also identify event-density and motion-related differences that affect interpretation on real sensors.

MouseSIS

  • MouseSIS evaluation uses all 25 publicly annotated clips, with “the mice” as the target and absent “a red car” as a negative control.The clips contain 4,946 annotated frames, and none is used for training.Source: E Real event-camera evaluation: details · p. 12

N-Caltech101

  • 0.52 mean IoU on N-Caltech101’s 200 samples, versus 0.37 for a wrong-class expression; the correct expression scores higher in 69% of samples.At 20 ms, the correct class expression reaches 0.57 mean IoU against the object contour.Source: E Real event-camera evaluation: details · p. 12

Real-event limitations

  • Real-event conditions differ from synthetic data: density shifts the useful window, hot pixels and background noise distort pseudo-RGB normalization, and stopped animals nearly vanish in short windows.MouseSIS hot pixels are removed using an event-count and local-neighbourhood rule.Source: E Real event-camera evaluation: details · p. 12

MouseSIS

  • Figure 3 shows real MouseSIS event inputs, predicted masks in red, and ground-truth contours in yellow for the mice and absent-car queries.The illustrated examples use a 75 ms window; the absent-car examples have empty predictions.Source: Figure 3 · p. 13

F Additional visualisations of the converted data

The figures visualize converted video and still-image event streams alongside event-density comparisons with real Gen1 recordings. They show events concentrated in moving regions for video and generated across still images by simulated saccades.

Event-density comparison

  • 4.9 vs. 8.1 ev/px/s are the medians for converted ChatUniVi clips and real Gen1 recordings, respectively.Figure 4 compares the distributions from 200 clips and real recordings.Source: Figure 4 · p. 13

Converted video examples

  • The 3-second ChatUniVi visualization pairs source frames with 100 ms event accumulations, where moving regions generate events and the static background stays silent.The event windows cover both original and FILM-upsampled frames.Source: Figure 5 · p. 14
  • The 0.2-second VidSTG visualization pairs two source frames with 100 ms event accumulations over original and optical-flow-upsampled frames.Source: Figure 6 · p. 14

Still-image conversion

  • A simulated ±8 px saccade converts a still image into ON/OFF events across the image extent, supplying training signal absent from a static image.Figure 7 encodes ON events in red and OFF events in blue.Source: Figure 7 · p. 14

Event-density comparison

  • Saccade-converted LLaVA-1.5 images have a median density of 40.8 ev/px/s, compared with 8.1 ev/px/s for real Gen1 recordings.The distributions cover 200 images and real Gen1 recordings.Source: Figure 8 · p. 15

Related papers

Loading 2610.39676v1…