PaperlayerView exact revision on arXiv

Source-linked AI summary

Quanta Perception as Probabilistic Events

Varun Sundar, Pavan Thodima, Sacha Jungerman, Mohit Gupta

arXiv:2608.27584v1cs.CVcs.AI

TL;DR

Quanta sensors capture photon-level information, but their data rates and reconstruction costs impede real-time perception in low-light and high-speed settings. This paper introduces recursive Bayesian probabilistic events, achieving kilohertz-scale outputs from streams exceeding 50,000 quanta frames per second, up to four orders of magnitude faster than reconstruction baselines.

  • Problem

    Photon-counting sensors provide abundant data, but their high-throughput streams and computationally intensive reconstructions remain poorly aligned with real-time robotic perception.

    Source: Introduction · p. 2

  • Method

    Probabilistic events maintain recursive per-pixel posteriors over time since the last intensity transition, producing motion-aware flux, temporal stability, and uncertainty signals.

    Source: Probabilistic Event Representations · p. 5

  • Results

    Up to four orders of magnitude faster than state-of-the-art reconstruction baselines, the pipeline processes over 50,000 quanta frames per second and produces kilohertz-scale outputs.

    Source: Introduction · p. 3

  • Takeaways & Limitations

    Probabilistic events provide a task-ready interface for low-latency quanta perception, supporting downstream vision in high-speed and low-light conditions without retraining vision models.

    Source: Introduction · p. 3, Low-Latency Quanta Perception in the… · p. 6

Abstract

from arXiv · show

Autonomous systems rely on extracting information from light, yet remain brittle in extreme environments, from nighttime navigation to high-speed robotics. Conventional sensors aggregate photons over fixed exposures, imposing trade-offs between sensitivity, dynamic range, and temporal resolution that degrade perception when photons are scarce or dynamics are rapid. Quanta sensors detect individual photons, but their streams exceed real-time compute and latency budgets by orders of magnitude. Here we introduce $\textit{probabilistic events}$, a computational primitive for real-time quanta perception from individual photon detections. By computing the posterior over the time since the last intensity change, we represent photon streams as recursive belief states. Rather than fixed-threshold event-camera triggers, this recursive Bayesian formulation yields three low-latency signals: motion-adaptive scene flux, high-fidelity activity maps, and entropy-based perceptual uncertainty. This representation enables perception in extreme conditions, including pose estimation of a running person at $\sim$0.05 lux---without retraining vision models. Our approach processes input streams exceeding 50{,}000 quanta frames per second on commodity GPU hardware---yielding kilohertz-scale outputs up to four orders of magnitude faster than state-of-the-art quanta reconstruction baselines, even for megapixel arrays. By replacing frame reconstruction with direct probabilistic inference over photon streams, this work bridges photon-counting quanta sensing with robotic vision.

Introduction

Probabilistic events address the information and latency bottlenecks that limit quanta sensing in extreme illumination and motion. Recursive Bayesian run-length inference produces uncertainty-aware, motion-adaptive representations that support real-time robotic perception without retraining vision models.

Introduction

  • 50,000+ qFPS input and kilohertz outputs up to four orders faster than reconstruction baselines enable pose estimation below 0.05 lux and nighttime geometric-semantic perception.The representation interfaces directly with off-the-shelf vision models without retraining.Source: Introduction · p. 3

Probabilistic Event Representations

  • Each pixel recursively maintains a posterior over run length—the time since the last intensity transition—forming a constant-time per-pixel belief state over temporal stability.The formulation replaces deterministic thresholded events with probabilistic inference over competing photon-arrival hypotheses.Source: Introduction · p. 2, Probabilistic Event Representations · p. 5
  • Three signals emerge: temporal stability maps, entropy changes, and motion-aware flux that adaptively suppress shot noise in static regions while limiting blur in dynamic areas.Stability encodes persistence across hundreds of microseconds to seconds, entropy tracks uncertainty evolution, and adaptive integration produces the flux estimate.Source: Probabilistic Event Representations · p. 5

Low-Latency Quanta Perception in the Wild

  • End-to-end perception reaches 4,000 FPS for edge detection, over 550 FPS for tracking, around 150 FPS for optical flow, and 1,000 FPS for low-light human pose estimation.These results span classical and neural vision tasks without retraining or task-specific tuning, and include representation generation plus downstream inference.Source: Low-Latency Quanta Perception in the… · p. 6, Figure 3 · p. 9

Comparison with Offline Quanta Image Reconstruction

  • Up to four orders higher throughput than reconstruction-first restoration methods enables causal, bounded-memory perception while preserving robust performance under fewer than 0.05 photons per pixel per frame.Probabilistic events adaptively integrate photons per pixel instead of performing dense spatiotemporal alignment.Source: Comparison with Offline Quanta Image… · pp. 10–11, Figure 4 · p. 10

Comparison with Conventional Low-Light and High-Speed Imaging

  • Probabilistic events retain usable intensity and temporal information where specialized cameras fail, enabling car detection at 100 Hz, QR decoding in 6 ms, depth at 100 Hz, and semantic segmentation.They avoid underexposure, motion blur, and the missing absolute intensity that limits conventional and polarity-event cameras.Source: Comparison with Conventional Low-Light… · pp. 11–12, Figure 5 · p. 12

B. Offline quanta-image restoration method A. Light level and motion extent

The section reports a 20 ms duration and 300 s runtime, with tracking performed using Segment Anything 2 across frames at 0, 38, and 76 ms.

B. Offline quanta-image restoration method

A. Light level and motion extent

C. Quanta probabilistic events (runtime: 0.62 ms; 400,000x faster)

Probabilistic events provide causal, low-latency representations from photon streams, while exposing fundamental information limits in extreme conditions. Their locally adaptive Bayesian states support near-sensor deployment and extend event-camera sensing with richer uncertainty-aware outputs.

C. Quanta probabilistic events (runtime: 0.62 ms; 400,000x faster)

  • 0.62 ms latency enables real-time probabilistic-event outputs, whereas Bit2Bit requires approximately 300 s per frame plus hours of sequence optimization, although rapid changes can blur.The method preserves discernible structure, and SAM 2 successfully tracks the object despite imperfect reconstruction.Source: Figure 6 · p. 13

Operating Limits at the Photon Limit

  • Reliability spans the middle 97% of photon detection probabilities, but extreme low light with rapid motion lowers signal-to-noise and causes motion to be absorbed into the integration window.Sparse photons can produce stale, overconfident posteriors and missed changes, while near-deterministic detections can trigger overly frequent changes; both degrade motion–shot-noise separation.Source: Operating Limits at the Photon Limit · p. 13

Toward Near-Sensor Probabilistic Processing

  • Bounded-state, per-pixel Bayesian recursion supports near-sensor processing, with the gradient variant sustaining approximately 3,200 qFPS at 1 megapixel on an NVIDIA Jetson Orin Nano.Fixed memory, constant-time updates, configurable aggregation, and causal local operation support bandwidth- and power-efficient task-ready perceptual fields for integrated sensing and inference.Source: Toward Near-Sensor Probabilistic… · p. 14

Relation to Event Cameras

  • Unlike threshold-triggered event cameras [36], probabilistic events maintain per-pixel change-interval posteriors that produce continuous activity, uncertainty, and motion-aware intensity fields.Their stateful outputs can be read at arbitrary cadence, retaining high-frequency internal updates while emitting image-like frames compatible with standard vision and compression pipelines.Source: Relation to Event Cameras · pp. 14–15

Methods … Spatially-Varying Denoising

The method converts high-rate photon detections into recursively inferred, motion-adaptive representations that balance noise, blur, uncertainty, spatial responsiveness, and throughput. It extends this core with bounded-memory inference, spatial features, direct perception, and spatially varying denoising for real-time quanta vision.

Image Formation in Quanta Cameras

  • Quanta cameras represent ≈100 kHz photon detections as binary Bernoulli frames, whose temporal sums form binomial virtual exposures but face a noise–blur trade-off under fixed integration.Summing 32 frames is noisy, whereas summing 8,000 introduces severe motion blur; explicit motion compensation is computationally prohibitive for real-time reconstruction.Source: Image Formation in Quanta Cameras · pp. 19–20

Computing Temporal Stability Maps

Estimating the Expected Adaptive Decay

  • Expected stability weights produce continuous adaptive decay that freezes fast motion with short exposures while accumulating 20 ms or more in static backgrounds.The expectation over discrete run-length hypotheses supports smooth intensity transitions and reduces ghosting through the updated weighting scheme.Source: Estimating the Expected Adaptive Decay · p. 22

Quantifying Uncertainty via Run-Length Entropy

  • Differential run-length entropy detects moving edges and corners through a rise in uncertainty followed by stabilization, complementing the intensity estimate with confidence information.The resulting entropy map highlights complex motion boundaries and low-confidence regions.Source: Quantifying Uncertainty via Run-Length… · p. 22

Extension to Binomial Frames

Runtime Analysis

  • ≈41,000 qFPS input and 1,280 FPS output are achieved at 1 megapixel by the fast binomial-plus-gradient variant, while robust filters remain viable for real-time operation.DoG reaches ≈16,000 qFPS and Log-Gabor exceeds 6,000 qFPS at 1 megapixel; the gradient variant sustains ≈3,200 qFPS on Jetson Orin Nano.Source: Runtime Analysis · p. 25

Quanta Perception Pipeline

  • The pipeline exposes motion-adaptive photon aggregates as a low-latency, plug-and-play input for detection, pose estimation, optical flow, and tracking without retraining or fine-tuning downstream models.It also supports optional visualization denoising and bandwidth-efficient transfer, avoiding the annotation and specialized-model requirements of quanta-specific training.Source: Quanta Perception Pipeline · p. 25, Direct Perception · pp. 25–26

Spatially-Varying Denoising

  • ∼4,000 FPS spatially varying Wiener filtering uses the temporal stability map to smooth short-integration, high-variance regions more aggressively while preserving temporal resolution.The method derives an effective run length for local noise estimation and avoids the throughput bottleneck of neural restoration, burst fusion, or optical-flow-guided merging.Source: Spatially-Varying Denoising · p. 26

Hardware-Accelerated Video Encoding … Code Availability

This section examines hardware-compatible quanta video transmission, color acquisition, causal temporal adaptation, and robustness to sensor non-idealities, alongside supplementary image-quality and throughput comparisons. It also describes the supporting datasets and code and notes that the images contain no identifiable human data.

Hardware-Accelerated Video Encoding

  • Approximately 6,000 FPS 256 × 512 streams are processed by GPU-based NVENC H.264 encoding, accommodating probabilistic-event output throughputs.The experiment compares delta-based COO transmission with direct encoding of denoised motion-aware flux aggregates using PSNR and SSIM rate-distortion curves.Source: Hardware-Accelerated Video Encoding · p. 27
  • One to two orders of magnitude higher compression ratios are achieved by video codecs than event-based transmission at comparable quality, expanding the bandwidth-design space.The choice depends on hardware topology: delta encoding is cheap for on-sensor compression, while standard codecs suit edge platforms or dedicated photon-processing units.Source: Hardware-Accelerated Video Encoding · p. 27

Color Quanta Perception

  • Probabilistic events extend to Bayer-mosaicked color acquisition by modeling temporal statistics on raw mosaics, extracting provisional luminance features, and applying channel-wise spatial denoising before debayering.The scalar temporal integration map is broadcast across spectral channels to maintain regularization under shared geometric confidence.Source: Color Quanta Perception · pp. 27–28

How Well Do We Estimate Temporal Stability?

  • 0.72 stops blur RMSE and 1.50 stops noise RMSE yield 1.28 stops overall stability RMSE, versus 1.39, 2.24, and 2.0 stops for the oracle global baseline.Log-Gabor spatial features suppress blur in dynamic regions while preserving integration time in static areas.Source: How Well Do We Estimate Temporal… · p. 29
  • Temporal stability estimation remains responsive but becomes evidence-limited at low light, while excessive quanta-frame aggregation shifts error toward motion blur.Increasing Nsum reduces under-exposure error only until the accumulated window exceeds local scene stability, making stability itself the local limit on useful evidence aggregation.Source: How Well Do We Estimate Temporal… · p. 29

Upper Bound on SNR Advantage

  • Per-pixel temporal windows exceed global exposure limits because global shutters are constrained by maximum scene velocity, whereas probabilistic events adapt integration inversely to local velocity.Static regions can integrate longer than dynamic foregrounds, improving SNR without introducing motion blur; the theoretical bound assumes constant photon probability and shot noise as the sole noise source.Source: Upper Bound on SNR Advantage · p. 30

Event-Driven Computer Vision

  • Probabilistic events provide causal, low-memory belief updates that jointly derive motion-adaptive intensity, activity, and uncertainty without explicit reconstruction, motion compensation, or domain-specific retraining.Spatial likelihoods exploit inter-pixel structure, improving the noise–blur trade-off over purely temporal Bernoulli formulations while binomial multi-bit measurements reduce computational overhead.Source: Computer Vision with Quanta Sensors · p. 32, Event-Driven Computer Vision · pp. 32–33

Quanta Sensor Non-Idealities

  • Real-data experiments include practical SPAD non-idealities, and stable operation across illumination and motion regimes suggests moderate departures from the Bernoulli model do not unduly affect the framework.The likelihood can be replaced with a calibrated model incorporating dark counts, dead time, saturation, or afterpulsing without changing the recursive Bayesian update.Source: Quanta Sensor Non-Idealities · pp. 35–36

Consent for Publication

  • The manuscript states that its images contain no identifiable human data, making consent for publication not applicable.Source: Consent for Publication · p. 42

Data Availability

  • The quanta sensor data supporting Figs. 1–5 and Extended Data Figs. 1, 4, 7, and 9 are openly available, while Fig. 6 data come from Liu et al..The VisionSim dataset used for Methods analysis is also available online.Source: Data Availability · p. 42

Code Availability

  • The supplementary code archive includes scripts that generate the manuscript’s quantitative tables and plots from the cited datasets.Code is provided as a compressed supplementary file.Source: Code Availability · p. 42

Extended Data

  • Within 0.7 dB of QUIVER, the denoised probabilistic event camera maintains > 1,000 FPS, balancing image quality with kilohertz-rate throughput.The comparison covers PSNR and SSIM at 0.05 PPP and 0.5 PPP; virtual exposures reach 10,000 FPS but lack denoising, while QUIVER operates below 1 FPS.Source: Extended Data · p. 43

A. Camera motion, ambient lighting (0.5 PPP) · B. Fast object motion, indoor lighting (0.6 PPP) · C. Moderate object motion, low light (0.05 PPP)

The supplied passage identifies streaming EMA video denoising for moderate object motion under low-light conditions at 0.05 PPP.

C. Moderate object motion, low light (0.05 PPP)

D. Output throughput (FPS)

Probabilistic events provide computationally lightweight outputs with fidelity comparable to or better than restoration and denoising baselines, while uniquely exceeding 1,000 FPS. Their motion-adaptive temporal windows preserve high-frequency details in fast-motion and low-light conditions.

Throughput

  • 33–2200× faster than the considered baselines, probabilistic events are the only method exceeding 1,000 FPS on desktop-grade CPU/GPU hardware.Runtime measurements used an Intel Xeon Silver CPU with an NVIDIA 4090 GPU accelerator.Source: D. Output throughput (FPS) · p. 44

Output quality

  • Probabilistic events provide output fidelity comparable to, and in some cases better than, QUIVER [20], FastDVDnet [25], and Streaming EMA [26].The comparison covers camera motion in ambient lighting, fast object motion indoors, and moderate object motion in low light.Source: D. Output throughput (FPS) · p. 44

Motion-aware aggregation

  • Shorter temporal windows assigned to dynamic regions reduce motion blur and preserve high-frequency details that learned baselines oversmooth in fast motion or low light.Examples include numbers and markings on a measuring tape.Source: D. Output throughput (FPS) · p. 44

A. Light level of 0.5 PPP

At 0.5 PPP, the section presents a comparison involving ground truth, exposure stacking, FastDVDNet, NAFNet, gQIR, Quanta burst, and probabilistic events.

B. Light level of 0.05 PPP

At 0.05 PPP, probabilistic events preserve downstream depth and flow performance while avoiding full reconstruction, and process megapixel quanta streams at high throughput. Their adaptive aggregation balances photon noise and motion blur, although extreme motion, abrupt changes, and compression can expose limitations.

Image quality

  • At low light, probabilistic events preserve sufficient scene structure for downstream inference at kilohertz output rates, despite reconstruction-first methods recovering higher-SNR images.At moderate light, they provide comparable PSNR and SSIM to state-of-the-art quanta restoration methods while operating up to four orders of magnitude faster.Source: B. Light level of 0.05 PPP · p. 45

Task performance

  • At 10× fewer photons and 5× faster motion, probabilistic events match reconstruction-first depth and optical-flow performance without full image restoration.At 20× fewer photons, depth remains robust, while optical flow becomes more sensitive to residual noise and extreme motion.Source: B. Light level of 0.05 PPP · pp. 46–47

Aggregation and spatial information

  • Adaptive exponential smoothing driven by temporal stability mediates the fixed-window noise–blur trade-off, but the pixel-wise formulation remains less responsive to abrupt changes.Spatial feature modeling improves selectivity in textured regions; Log-Gabor features are more selective, while gradients provide a lighter computational alternative.Source: B. Light level of 0.05 PPP · p. 48

Throughput

  • Up to four orders of magnitude faster than reconstruction-first baselines, probabilistic events process megapixel streams at 41,000 qFPS and emit up to 1,286 FPS on an RTX 4090.On Jetson Orin Nano, variants sustain approximately 8,700 and 3,200 qFPS at 1 MPixel, corresponding to approximately 270 and 100 FPS output.Source: B. Light level of 0.05 PPP · p. 49

Compression

  • 2660× compression at 27 dB PSNR outperforms event-based encoding’s 50× compression at 21 dB, which exhibits ghosting during camera panning.The comparison indicates that event-based compression preserves structural integrity less effectively than H.264 at this extreme setting.Source: B. Light level of 0.05 PPP · p. 50

Color imaging

  • High-fidelity color imaging is obtained by aggregating photons to the per-pixel noise–blur limit, followed by demosaicking and Wiener denoising.The representation supports color capture across rapid swiveling, falling objects, and nighttime driving scenes.Source: B. Light level of 0.05 PPP · p. 51

A. Scene with planar ego motion: one dominant pixel velocity

This section compares probabilistic-event variants with groundtruth and best global exposure references, reporting blur and noise RMSE differences in stops.

A. Scene with planar ego motion: one dominant pixel velocity

B. Scene with few distinct pixel velocities

This section compares ground truth, best global exposure, and probabilistic-event variants, including binomial, max-filtered, and Log-Gabor formulations.

B. Scene with few distinct pixel velocities

C. Estimation error across the dataset

Across the dataset, probabilistic events estimate temporally stable exposures by adapting integration windows to local motion and light, reducing blur while controlling photon-limited noise. Increasing temporal aggregation improves evidence in low light but eventually sacrifices responsiveness through motion blur.

C. Estimation error across the dataset

  • 0.26 stops blur RMSE is achieved in distinct-velocity scenes, versus 1.68 stops for the oracle global shutter, as probabilistic events assign integration windows locally.The method uses short windows for falling cards and long windows for the background, while the oracle is forced into a global compromise.Source: C. Estimation error across the dataset · p. 52
  • 0.58 stops blur RMSE versus 1.01 stops for the oracle is achieved under uniform ego-motion, while aggregate results show improved blur and noise metrics over the oracle.Minimum filtering removes blur but produces excessive noise, whereas Log-Gabor balances the noise–blur trade-off.Source: C. Estimation error across the dataset · p. 52
  • As light decreases, larger summed-frame counts reduce noise but eventually increase blur, exposing a trade-off between photon aggregation and temporal responsiveness.The optimal summed-frame count is bounded by scene stability: low counts remain responsive but evidence-limited, while excessive aggregation blurs moving regions.Source: C. Estimation error across the dataset · pp. 53–54
  • Motion-aware per-pixel integration improves SNR over a global shutter by allowing static regions to integrate longer than the maximum global exposure permits.The global exposure is constrained by maximum scene velocity, so pixels with faster dynamics than the minimum exposure incur inevitable motion blur.Source: C. Estimation error across the dataset · p. 55

Related papers

Loading 2608.27584v1…