PaperlayerView exact revision on arXiv

Source-linked AI summary

Event-guided Neural Video Compression

Jiyun Kong, Jungwoo Kim, Enes Eray Demirtas, Touradj Ebrahimi, Jong-Seok Lee

arXiv:2610.02265v1eess.IVcs.CVcs.MM

TL;DR

RGB-only temporal contexts can be less reliable under diverse motion or long prediction chains, potentially requiring more RGB bitrate. ENVC uses shared events for motion coding and event-conditioned temporal context refinement; across six benchmarks, it reports average BD-rate savings of 39.13% using PSNR-RGB and 67.63% using LPIPS relative to DCMVC.

  • Problem

    RGB-only temporal contexts can become less reliable under diverse motion or long prediction chains, potentially requiring more RGB information to maintain reconstruction quality.

    Source: §1 · p. 1

  • Method

    ENVC uses shared events to form a motion prior and code residual motion, while an event-conditioned predictor supplies features for gated temporal context refinement.

    Source: §1 · p. 2

  • Results

    39.13% using PSNR-RGB and 67.63% using LPIPS average BD-rate savings relative to DCMVC across six benchmarks paired with synthetic events.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The results suggest shared events are a promising complementary modality for RGB video compression and motivate research on efficient event coding and joint RGB–event compression.

    Source: §6 · p. 10, §7 · p. 10

    The current implementation adds computational overhead because event-conditioned prediction and motion estimation run at both encoder and decoder.

    Source: §6 · p. 10

Abstract

from arXiv · show

Neural video codecs derive motion and temporal contexts mainly from RGB frames, leaving room for cross-modal guidance from complementary temporal observations. Event streams can provide such observations by recording brightness changes between frames. In this work, we propose an Event-guided Neural Video Codec (ENVC) that uses events shared by the encoder and decoder to improve RGB compression efficiency. For motion coding, ENVC forms an event-guided motion prior and codes the remaining motion residual. For frame coding, an event-conditioned predictor supplies multi-scale features for gated temporal context refinement. To support training and evaluation on standard video datasets, we synthesize paired RGB-event data and assess its predictive utility through comparisons with real events. Across six benchmarks, ENVC achieves average BD-rate savings of 39.13% using PSNR-RGB and 67.63% using LPIPS relative to DCMVC. Further analyses show that our gains persist on large-motion sequences and that ENVC effectively learns to integrate event information. These results demonstrate the potential of events as a complementary modality for reducing the RGB coding rate. Our model and code are available at https://github.com/kjungwoo03/ENVC.

1 INTRODUCTION

The introduction argues that RGB-only temporal contexts may become less reliable with diverse motion or long prediction chains, potentially requiring more RGB information, and asks whether shared events can improve contexts and reduce bitrate. ENVC uses events in motion coding and temporal context refinement, with reported gains across six benchmarks.

1 INTRODUCTION

  • RGB-only contexts rely on decoded frames and propagated features, which may be unreliable under diverse motion or long prediction chains and require more RGB information.The paper asks whether complementary observations of inter-frame changes can improve contexts and reduce the required RGB bitrate.Source: §1 · p. 1
  • Events asynchronously record inter-frame brightness changes; their polarity indicates change direction, and their sensing offers microsecond-level temporal resolution and high dynamic range.An event is triggered when the magnitude of a pixel’s log-intensity change exceeds a contrast threshold.Source: §1 · p. 1
  • Unlike event-only reconstruction, ENVC shares events at both coding ends while retaining current-frame RGB information to improve reconstruction quality.Event-only prediction can be limited by reconstruction errors; ENVC instead codes additional current-frame RGB information under a rate–distortion objective.Source: §1 · p. 2, Figure 1 · p. 2
  • ENVC forms an event-guided motion prior and codes the remaining motion residual; gated event-conditioned features refine temporal contexts for frame coding.The event-conditioned predictor also supplies a frame prediction used in context refinement.Source: §1 · p. 2
  • 39.13% PSNR-RGB and 67.63% LPIPS average BD-rate savings relative to DCMVC across six benchmarks, with gains persisting on large-motion sequences.The evaluation uses synthetic events under the shared-event setting.Source: §1 · p. 2

2 RELATED WORK

Related work advances neural video coding through temporal-context modeling and combines events with frames for reconstruction. ENVC positions shared events as temporal side information for motion coding and context construction.

2 RELATED WORK

  • Neural video codecs use learned motion and residual coding or temporal-context-conditioned frame coding, with context modeling advanced through multi-scale mining, reference modulation, and multi-reference aggregation.These approaches complement RGB-derived temporal information with shared events in ENVC.Source: §2 · p. 3
  • Event-based reconstruction can use only events, while frame-guided methods combine images and events for interpolation or single-reference prediction; ENVC codes additional current-frame RGB information under a rate–distortion objective.The passage notes that events do not directly measure absolute intensity and that single-reference prediction is compatible with sequential decoding.Source: §2 · p. 3
  • ENVC investigates shared event observations as temporal side information for motion coding and temporal context construction, extending cross-modal guidance to events.The cited prior compression methods use visible–infrared, RGB–depth, geometric, semantic, or depth information.Source: §2 · p. 3

3 SYNTHETIC RGB–EVENT DATASET CONSTRUCTION

The section constructs paired synthetic events for standard video datasets, aligns events to original-frame coding intervals, and evaluates whether synthetic events provide temporal guidance comparable to real events.

3 SYNTHETIC RGB–EVENT DATASET CONSTRUCTION

  • Because standard NVC datasets lack paired events, the authors synthesize events from RGB datasets to enable rate–distortion comparisons on the same video content.The original RGB frames remain the coding frames.Source: §3 · p. 3

3.1 EVENT SYNTHESIS PIPELINE

  • Super-SloMo densifies videos to a nominal rate of at least 200 fps; grayscale intermediates feed DVS-Voltmeter, and events are binned between original-frame timestamps.The original RGB frames are retained for coding, while densified frames are used only for event synthesis.Source: §3.1 · pp. 3–4

3.2 SYNTHETIC EVENT QUALITY ASSESSMENT

  • The assessment compares event counts and spatial distributions, then measures VFPSIE PSNR-RGB using real or synthetic events against a reference-only baseline.It evaluates 31 adjacent intervals within the first 32 frames of 15 HS-ERGB test sequences and 17 BS-ERGB validation sequences.Source: §3.2 · p. 4
  • 1.54 and 1.59 times as numerous as real events on HS-ERGB and BS-ERGB, respectively, synthetic events showed spatial agreement and enabled comparable VFPSIE prediction quality.Qualitative examples show similar dominant spatial structures, including object contours and concentrated event activity.Source: §3.2 · p. 4

4 PROPOSED METHOD

ENVC uses shared events to guide motion coding and refine temporal contexts for conditional frame coding. Its pipeline combines event-based prediction, residual motion coding, and multi-scale context refinement.

4.1 FRAMEWORK OVERVIEW

  • Both encoder and decoder use the reconstructed reference and shared events; EFP predicts a frame and multi-scale contexts that condition frame coding, entropy modeling, and reconstruction.Source: §4.1 · p. 5

4.2 EVENT-PREDICTIVE MOTION CODING

  • RMC codes the difference between event-based and current-frame motion estimates, then adds the decoded residual to the event-based estimate to reconstruct motion.Source: §4.1 · p. 5, §4.2 · p. 6

4.3 EVENT-GUIDED CONTEXT REFINEMENT

  • ECR backward-warps multi-scale reference features with reconstructed motion, injects EFP contexts through gated residuals, and PCF adds prediction features to the ×1 context.Source: §4.3 · p. 6

4.4 FRAME CODING AND OPTIMIZATION

  • The conditional DCVC-DC frame codec encodes and reconstructs frames using temporal contexts; the ×1/4 context also serves as an entropy-model prior.Source: §4.4 · p. 6
  • The main rate–distortion term balances rate with reconstruction distortion, with λ controlling the trade-off.Source: §4.4 · p. 7

5 EXPERIMENTS

Experiments compare ENVC with neural video codecs on six benchmarks and analyze its rate–distortion performance, event integration, motion sensitivity, intra periods, and computational cost. ENVC shows substantial average BD-rate savings, while results also expose quality trade-offs and implementation overhead.

5.1 EXPERIMENTAL DETAILS

  • ENVC evaluation uses synthetic-event pairs, reporting PSNR-RGB and LPIPS BD-rate relative to DCMVC, with coding rates in bpp.Training uses Vimeo-90K paired with synthetic events; evaluation covers UVG, MCL-JCV, and HEVC Classes B–E, generally on 96 frames with intra period 32.Source: §5.1 · p. 7

5.2 RATE–DISTORTION PERFORMANCE

  • Across six benchmarks, ENVC saves 39.13% BD-rate using PSNR-RGB and 67.63% using LPIPS relative to DCMVC.Figure 6 plots LPIPS against bitrate on the six benchmarks.Source: Figure 6 · p. 8, §5.2 · p. 8
  • On HEVC-E, PSNR-RGB savings are 7.76%, while LPIPS-based savings are 51.60%.The text associates the smaller PSNR-RGB gain with limited inter-frame motion and a 56.2% I-frame share of total RGB rate at Q0.Source: §5.2 · p. 8

5.3 HOW DOES ENVC USE EVENTS?

  • Mean event-gate activations are 0.373, 0.140, and 0.130 across the ×1, ×1/2, and ×1/4 scales; fixing gates to 0 or 1 lowers PSNR by 2.30–3.89 dB.Gating lowers adapter-feature RMS ratios below reference-context magnitudes while retaining event-conditioned signals; mean gate activation alone does not characterize injection magnitude.Source: §5.3 · pp. 8–9, Figure 7 · p. 8

5.4 EFFECTS OF MOTION MAGNITUDE AND INTRA PERIOD

  • BD-rate savings are 56.60% for large-motion sequences versus 47.96% for small-motion sequences, with substantial savings across all four P-frame motion quartiles.Quartile savings from lowest to highest motion are 40.8%, 69.3%, 74.8%, and 65.9%.Source: §5.4 · p. 9, Figure 8 · p. 9
  • Increasing the intra period from 32 to 64, 96, and 192 frames reduces mean bpp by 25.7%, 33.3%, and 37.7%, with PSNR-RGB losses of 0.37, 0.77, and 1.21 dB.The largest incremental rate saving comes from increasing the period to 64; further increases yield smaller additional savings while quality continues to decline.Source: §5.4 · p. 9, Figure 8 · p. 9

5.5 COMPLEXITY ANALYSIS

  • ENVC has 33.52M parameters and runs at 0.323 encoding and 0.386 decoding FPS on a single RTX 3090; event modules add computational cost to both coding stages.Measurements use the 1920×1080 UVG Beauty sequence.Source: §5.5 · p. 9, Table 3 · p. 9

6 DISCUSSION

The discussion frames events as a complementary modality for RGB compression, while identifying computational overhead as a constraint on the current implementation.

6 DISCUSSION

  • ENVC is an initial exploration focused on event integration into motion coding and temporal context construction, with event-conditioned prediction and motion estimation adding computation at both ends.The authors identify lightweight guidance modules and efficient pipelines that reduce complexity and end-to-end latency as future work.Source: §6 · p. 10

7 CONCLUSION

The conclusion presents ENVC as a shared-event approach to RGB compression, with benchmark gains and additional analyses of practical use and event quality. The paper also describes application settings where events are already available and outlines training and evaluation choices.

7 CONCLUSION

  • Across six synthetic-event benchmarks, ENVC saves 39.13% BD-rate using PSNR-RGB and 67.63% using LPIPS relative to DCMVC.The conclusion describes shared event observations as a complementary modality for reducing RGB coding rate.Source: §7 · p. 10

A PRACTICAL SCENARIOS

  • When events already reach a server for perception, a decoder can reuse them while RGB remains available for human inspection and interpretation.The paper gives robotics and industrial systems as examples of this setting.Source: A PRACTICAL SCENARIOS · p. 15

B TRAINING DETAILS

  • The event-compression test uses a fixed checkpoint on seven UVG sequences; training uses five stages and cascaded sequences extended to 32 frames.Training data include 9,220 paired synthetic-event clips; the test varies event quality without fine-tuning.Source: §A.1 · p. 15, B TRAINING DETAILS · p. 16

A.1 SENSITIVITY TO EVENT COMPRESSION

  • With coordinate quantization at qE = 0.5, 99.9% of events remain but occupied pixels fall to 53.4% and count-map correlation is 0.396; PSNR-RGB reaches 30.70 dB.The authors report nonmonotonic performance: qE = 0.02 yields 32.69 dB despite poorer event fidelity, and suggest the mechanism is not isolated.Source: §A.1 · pp. 15–16, Figure 9 · p. 16

C SEQUENCE-LEVEL GAINS AND FAILURES

ENVC improves PSNR-RGB coding on most examined sequences, though gains vary with content and coding conditions. It also operates on real sensor events without fine-tuning, but performance is lower than with synthetic events.

C SEQUENCE-LEVEL GAINS AND FAILURES

  • ENVC achieves negative PSNR-RGB BD-rates on 50 of 53 sequences, with a median of −44.9% relative to DCMVC.The analysis covers the first 96 frames with an intra period of 32.Source: C SEQUENCE-LEVEL GAINS AND FAILURES · p. 17
  • Event density correlates with savings (Spearman ρ = −0.47), but does not determine them: UVG Beauty saves 82.4% at only 0.146 events per pixel.The 13 sequences with largest savings average −71.1% BD-rate versus −7.1% for the 13 with smallest savings.Source: C SEQUENCE-LEVEL GAINS AND FAILURES · p. 17
  • Jockey has the largest PSNR-RGB loss (+56.6% overall), yet its LPIPS BD-rate is −52.4%, showing that its relative outcome depends on the metric.Complex motion, occlusions, and repetitive textures are suggested as possible sources of difficulty, but their contributions are not isolated.Source: C SEQUENCE-LEVEL GAINS AND FAILURES · p. 17
  • On low-activity HEVC-E, ENVC’s P-frame-only PSNR-RGB BD-rate is −32.0%, while its overall saving is −7.8%.At Q0, I-frames account for 56.2% of ENVC’s total RGB rate versus 30.1% for DCMVC; ENVC also uses 66% fewer P-frame bits, with mean P-frame PSNR of 37.71 dB versus 38.80 dB.Source: C SEQUENCE-LEVEL GAINS AND FAILURES · p. 17

D GENERALIZATION TO REAL EVENTS

  • With real sensor events, ENVC has higher RGB rates, lower PSNR-RGB, and higher LPIPS than with synthetic events on both datasets, although it operates without fine-tuning.Synthetic events are denser on average, but density alone does not determine coding gains; differences in event statistics and training-distribution compatibility may also contribute.Source: D GENERALIZATION TO REAL EVENTS · p. 18

E ANALYSIS OF LEARNED EVENT GATING

Learned event gates vary more across space than across time, with distinct spatial patterns across scales. Interventions indicate benefits from aligned events, while fixed-gate tests alter operating points without isolating spatial variation.

E ANALYSIS OF LEARNED EVENT GATING

F LEARNED USE OF SPATIOTEMPORALLY ALIGNED EVENTS

  • Aligned events support higher reconstruction quality at lower coding rates; a one-frame shift reduces PSNR-RGB by 0.72–3.79 dB and yields mean per-sequence bpp ratios of 1.99–9.86.Both shuffle conditions perform worse than zero-event inputs in PSNR and bpp on every evaluated sequence.Source: Figure 13 · p. 20, Table 7 · p. 20, F LEARNED USE OF SPATIOTEMPORALLY… · p. 20
  • Event reliance varies by content: removing or temporally reassigning events has smaller effects on selected HEVC-E sequences than on MCL-JCV examples.Increasing event delay also produces sequence-dependent changes.Source: F LEARNED USE OF SPATIOTEMPORALLY… · p. 20

G BIT ALLOCATION ANALYSIS

On UVG, ENVC reduces both motion and conditional frame coding rates, with most P-frame bit savings coming from frame coding. Savings are concentrated in blocks with more event activity, although the block comparison does not match reconstruction quality.

G BIT ALLOCATION ANALYSIS

  • The event-count block comparison is at Q1, where ENVC and DCMVC mean PSNR-RGB values are 35.71 and 36.40 dB, respectively, so reconstruction quality is not matched.The analysis groups 16×16 P-frame blocks by event count.Source: G BIT ALLOCATION ANALYSIS · p. 20
  • Over the common quality range, ENVC reduces motion coding rate by 67.9% and conditional frame coding rate by 39.8%.Frame coding accounts for 86.4% of P-frame bit savings; matching average quality selects higher ENVC settings and partially offsets P-frame savings through increased shared intra-codec rate.Source: G BIT ALLOCATION ANALYSIS · p. 20
  • In blocks without events, ENVC’s frame-latent rate is 53% of DCMVC’s; in blocks with at least 64 events, it is 15–18%.Blocks with at least 16 events make up 35.4% of blocks but account for 69.6% of frame-latent bit savings.Source: G BIT ALLOCATION ANALYSIS · p. 20

H ADDITIONAL DATASETS AND QUALITY METRICS

Additional evaluations show ENVC’s advantages on 4K video across the reported quality measures and across additional metrics on six benchmarks.

H ADDITIONAL DATASETS AND QUALITY METRICS

I ADDITIONAL QUALITATIVE RESULTS

Additional qualitative results compare real and synthetic event inputs, ENVC and DCMVC reconstructions at similar RGB rates, and event activity alongside RGB frame changes. They show spatial discrepancies between event sources, better preservation of local textures and object boundaries by ENVC in selected examples, and event activity concentrated around spatio-temporal changes.

Real and Synthetic Event Comparison

Qualitative Reconstruction Comparison

  • At similar RGB coding rates, ENVC better preserves local textures and object boundaries than DCMVC in selected examples.The examples complement the quantitative rate–distortion results; reported PSNR and bpp are measured over the full frame.Source: I ADDITIONAL QUALITATIVE RESULTS · p. 21, Figure 20 · p. 25

Frame Changes and Event Activity

  • Event activity is concentrated around spatio-temporal changes between consecutive RGB frames in the illustrated sequences.The examples pair previous and target frames with their frame-difference maps and synthetic events for the corresponding interval.Source: I ADDITIONAL QUALITATIVE RESULTS · p. 21, Figure 21 · p. 26

Additional Quality Metrics

  • Additional rate–distortion comparisons cover PSNR-Y and MS-SSIM across six benchmarks, while Table 9 reports BD-rate against DCMVC for both metrics.The table caption states that negative BD-rate values indicate bitrate savings; no cell values are given in these passages.Source: Table 9 · p. 22, Figure 16 · p. 22, Figure 17 · p. 22

Related papers

Loading 2610.02265v1…