Source-linked AI summary
Quanta Perception as Probabilistic Events
Varun Sundar, Pavan Thodima, Sacha Jungerman, Mohit Gupta
TL;DR
Autonomous vision loses information under low light and rapid motion, while quanta sensors create streams too demanding for real-time processing. The paper introduces recursive Bayesian probabilistic events over photon-stream run lengths, producing uncertainty-aware task-ready signals. It processes more than 50,000 qFPS with kilohertz-scale outputs up to four orders of magnitude faster than reconstruction baselines and supports downstream perception without retraining.
Problem
Fixed-exposure cameras degrade perception in scarce-photon or rapidly changing scenes, while quanta streams exceed practical real-time compute and latency budgets.
Method
Probabilistic events recursively maintain per-pixel Bayesian posteriors over time since the last intensity transition and derive motion-aware flux, temporal stability, and entropy signals.
Results
More than 50,000 qFPS input and kilohertz-scale output run up to four orders of magnitude faster than reconstruction baselines while supporting perception without retraining.
Takeaways & Limitations
Probabilistic events provide a direct interface from photon streams to off-the-shelf vision models for low-latency perception in extreme light and motion conditions.
Abstract
from arXiv · showhide
Autonomous systems rely on extracting information from light, yet remain brittle in extreme environments, from nighttime navigation to high-speed robotics. Conventional sensors aggregate photons over fixed exposures, imposing trade-offs between sensitivity, dynamic range, and temporal resolution that degrade perception when photons are scarce or dynamics are rapid. Quanta sensors detect individual photons, but their streams exceed real-time compute and latency budgets by orders of magnitude. Here we introduce $\textit{probabilistic events}$, a computational primitive for real-time quanta perception from individual photon detections. By computing the posterior over the time since the last intensity change, we represent photon streams as recursive belief states. Rather than fixed-threshold event-camera triggers, this recursive Bayesian formulation yields three low-latency signals: motion-adaptive scene flux, high-fidelity activity maps, and entropy-based perceptual uncertainty. This representation enables perception in extreme conditions, including pose estimation of a running person at $\sim$0.05 lux---without retraining vision models. Our approach processes input streams exceeding 50{,}000 quanta frames per second on commodity GPU hardware---yielding kilohertz-scale outputs up to four orders of magnitude faster than state-of-the-art quanta reconstruction baselines, even for megapixel arrays. By replacing frame reconstruction with direct probabilistic inference over photon streams, this work bridges photon-counting quanta sensing with robotic vision.
Introduction
The paper introduces probabilistic events to address information loss in conventional sensing and the computational burden of quanta photon streams. Recursive Bayesian belief states produce task-ready, uncertainty-aware representations for low-latency perception across extreme light and motion conditions.
- Conventional cameras lose continuous-time photon dynamics through fixed exposure windows, entangling sensitivity, noise, motion blur, and dynamic range.
- Quanta sensors preserve individual photon arrivals but generate data volumes and reconstruction costs that exceed practical robotic bandwidth, compute, and latency budgets.
- Probabilistic events recursively infer each pixel’s run-length posterior, replacing deterministic threshold triggers with a belief state over intensity change and uncertainty.
- Three derived signals—motion-aware photon aggregates, temporal stability maps, and entropy shifts—encode adaptive flux, activity timescales, and uncertainty evolution.
- 50,000+ qFPS input and kilohertz-scale output run up to four orders of magnitude faster than reconstruction baselines, enabling off-the-shelf models without retraining.
- The representation supports high-speed and low-light tasks, including 4,000 FPS edge detection, runner pose estimation, and robust perception under sparse photons and motion.
- Compared with offline reconstruction and specialized cameras, probabilistic events adapt integration locally, preserving dynamic structure while supporting low-light detection, QR decoding, depth, and segmentation.
B. Offline quanta-image restoration method A. Light level and motion extent
The supplied material reports a 20 ms duration and a 300 s runtime for tracking using Segment Anything 2.
- 20 ms is the reported duration for the displayed sequence.
- 300 s is the reported runtime associated with the displayed sequence.
- The material labels the task as tracking using Segment Anything 2.
- The displayed timeline marks t = 0 ms, t = 38 ms, and t = 76 ms.
C. Quanta probabilistic events (runtime: 0.62 ms; 400,000x faster)
Probabilistic events use bounded-state Bayesian updates to adapt photon integration locally while producing task-ready, uncertainty-aware visual fields at low latency. The approach supports near-sensor processing and flexible output cadence, but remains constrained when photon statistics cannot distinguish motion from shot noise.
- Operating limits: Extreme photon sparsity or near-deterministic detections degrade motion-versus-noise separation, causing stale posteriors, missed changes, or over-frequent change declarations.The estimator is most reliable across the middle 97% of photon detection probabilities per quanta frame; extreme low light with rapid motion can produce blur.
- Adaptive integration: Per-pixel Bayesian updates adapt integration timescales locally, allowing static regions to accumulate evidence while contracting integration around dynamic structure.The estimator performs causal, constant-time updates with bounded memory.
- Near-sensor deployment: Direct inference can transmit task-ready perceptual fields instead of raw binary detections, reducing bandwidth and power demands while shifting computation toward higher-level perception.The authors describe this as a path toward a software-defined quanta camera and integrated sensing-and-inference systems.
- Representation: Probabilistic events replace thresholded event triggers with per-pixel posterior distributions over temporal stability and photon-derived uncertainty.The representation provides continuous activity, entropy, and motion-aware flux signals rather than binary spikes.
- Output architecture: Stateful internal updates can run at high frequency while image-like outputs are read out at arbitrary, task-appropriate cadences.This decouples internal update rate from output rate and permits compatibility with mature video-compression pipelines.
Methods
The method converts high-speed photon detections into adaptive visual streams using probabilistic inference, balancing denoising, motion responsiveness, and real-time throughput. Spatial and temporal extensions provide task-ready outputs without requiring explicit motion compensation or retraining downstream vision models.
- Quanta image formation: Binary photon frames are summed into virtual exposures, producing binomial statistics and enabling maximum-likelihood scene-flux estimation under constant intensity.The binomial model assumes independent, identically distributed Bernoulli detections during the selected exposure.
- Quanta image formation: 32 frames produce noisy images, whereas 8000 frames introduce severe motion blur, exposing the fixed-exposure noise–blur compromise.Motion compensation can mitigate this trade-off but is computationally prohibitive for real-time reconstruction.
- Probabilistic inference: Probabilistic events estimate temporal stability from photon histories, enabling adaptive smoothing and denoising without explicit motion alignment.Expectation over run-length hypotheses yields a continuous stability metric that can represent smooth intensity transitions.
- Probabilistic inference: Entropy differences detect moving edges and corners through a rise in uncertainty followed by stabilization as the estimator converges.This produces a characteristic leading front of rising uncertainty and wake of stabilization.
- Throughput and deployment: The binomial formulation reaches ≈150,000 qFPS from 4-bit frames formed by summing 15 binary frames, with minimum exposure times of approximately 0.1–0.3 ms.The throughput increase may introduce minor motion blur, but the resulting exposure remains ultra-fast for many applications.
- Throughput and deployment: The fast binomial-plus-gradients variant achieves ≈41,000 qFPS input and 1,280 FPS output for a 1 MPixel sensor, while the gradient variant reaches ≈3,200 qFPS at 1 MPixel on Jetson Orin Nano.These results establish real-time operating points on both desktop-grade and power-constrained hardware.
Hardware-Accelerated Video Encoding
Probabilistic events support low-latency video output and adaptive sensing by estimating temporal stability per pixel, while their observation model remains an idealization of detector behavior.
- Hardware-Accelerated Video Encoding: 6,000 FPS: NVENC H.264 processes 256 × 512 video streams, accommodating probabilistic-event output throughputs.The low-latency image representations are compatible with hardware-accelerated video codecs.
- Hardware-Accelerated Video Encoding: One to two orders of magnitude: video codecs achieve higher compression ratios than event-based approaches at comparable quality.The preferred compression strategy depends on hardware topology and available computational budgets.
- Adaptive Temporal Integration: 1.68 stops: a global compromise produces high blur penalty in card-shuffling scenes with distinct independent velocities.Probabilistic events assign short windows to falling cards and longer integration to the static background.
- Temporal Stability Estimation: Roughly one photographic stop: probabilistic events recover temporal stability from noisy quanta measurements.More aggregation helps in low light until the summed window exceeds local scene stability, after which motion blur increases.
- Adaptive Temporal Integration: Per-pixel temporal stability lets static backgrounds integrate longer than dynamic foregrounds, maximizing SNR without motion blur.Global exposure must instead be constrained by the scene’s fastest dynamics.
- Model Scope: Idealized photon-counting assumptions omit detector effects including dead time, dark counts, afterpulsing, crosstalk, saturation, and pile-up.The authors describe Bernoulli likelihood as tractable rather than physically complete, while real-data experiments include sensor non-idealities.
C. Moderate object motion, low light (0.05 PPP)
The section concerns moderate object motion under low-light conditions at 0.05 PPP.
- C. Moderate object motion, low light (0.05 PPP): The section addresses object motion in a low-light setting.Its title specifies moderate motion.
D. Output throughput (FPS)
The section compares output throughput across reconstruction and probabilistic-event approaches.
- D. Output throughput (FPS): Probabilistic events are compared with ground truth and alternative reconstruction methods.Listed methods include Exposure stack, FastDVDNet, NAFNet, gQIR, and Quanta burst.
- D. Output throughput (FPS): Output throughput is the focus of the section.The section title identifies FPS as the measured quantity.
- D. Output throughput (FPS): The comparison includes Groundtruth, Exposure stack, FastDVDNet, NAFNet, gQIR, and Quanta burst.These entries are presented as comparison labels.
A. Light level of 0.5 PPP
The section concerns image-quality comparisons at a light level of 0.5 PPP.
- A. Light level of 0.5 PPP: Image-quality comparison methods include Groundtruth, Exposure stack, FastDVDNet, and NAFNet.These methods are listed for the section’s comparison.
- A. Light level of 0.5 PPP: The comparison also includes gQIR, Quanta burst, and Probabilistic events.These are listed as additional comparison entries.
B. Light level of 0.05 PPP
At 0.05 PPP, probabilistic events are evaluated against fixed exposures and reconstruction-first methods for image quality, throughput, and downstream perception. The representation uses adaptive temporal stability to balance photon-starvation noise against motion blur.
- B. Light level of 0.05 PPP: At 0.05 PPP, probabilistic events are compared with fixed-window exposures and reconstruction methods across image-quality and task-performance measures.The evaluation includes PSNR, SSIM, monocular depth, and optical flow.
- B. Light level of 0.05 PPP: At 20× fewer photons and 5× faster motion, probabilistic events retain robust depth performance, while optical flow becomes more sensitive to residual noise and extreme motion.They remain substantially better than fixed-window virtual exposures in this setting.
- B. Light level of 0.05 PPP: 32 summed quanta frames preserve motion but remain photon-starved, whereas 8000 summed frames reduce noise while introducing motion blur.Probabilistic-event representations include temporal stability maps, significant entropy changes, and motion-aware flux aggregates.
- B. Light level of 0.05 PPP: 41,000 qFPS input and 1,286 FPS output are achieved by the Gradient configuration on an RTX 4090 at 1 MPixel.On a Jetson Orin Nano, the Binomial and Gradient variants sustain approximately 8,700 and 3,200 qFPS at 1 MPixel.
- B. Light level of 0.05 PPP: 2660× compression at 27 dB PSNR is achieved by H.264, compared with 50× compression at 21 dB PSNR for event-based encoding.The event-based encoding exhibits laggard ghosting during camera panning, while H.264 better preserves structural integrity.
A. Scene with planar ego motion: one dominant pixel velocity
For a scene with planar ego motion and one dominant pixel velocity, probabilistic events are evaluated against the best global exposure using blur and noise error in stops.
- A. Scene with planar ego motion: one dominant pixel velocity: The compared probabilistic-event configurations are binomial, max filtering, and Log Gabor.These configurations are listed alongside ground truth and best global exposure.
- A. Scene with planar ego motion: one dominant pixel velocity: 1.01 stops blur RMSE and 1.48 stops noise RMSE quantify the reported exposure errors for the comparison.The comparison includes ground truth, best global exposure, and probabilistic-event variants.
- A. Scene with planar ego motion: one dominant pixel velocity: The evaluation reports error in stops using separate blur RMSE and noise RMSE quantities.The two reported values distinguish motion-blur and noise-related exposure error.
B. Scene with few distinct pixel velocities
This section presents a comparison involving ground truth, best global exposure, and probabilistic-event representations for a scene with few distinct pixel velocities.
- B. Scene with few distinct pixel velocities: Ground truth and best global exposure are included as comparison references.
- B. Scene with few distinct pixel velocities: Probabilistic-event representations include binomial, max filtering, and Log Gabor variants.
- B. Scene with few distinct pixel velocities: The section’s displayed comparison contains additional encoded entries alongside the named references and probabilistic-event variants.
C. Estimation error across the dataset
Across light levels and summed-frame counts, probabilistic-event temporal-stability estimates exhibit a noise–blur trade-off: more aggregation improves evidence but excessive aggregation blurs motion.
- C. Estimation error across the dataset: 0.75 stops blur RMSE and 2.12 stops noise RMSE are reported for another evaluated condition.
- C. Estimation error across the dataset: 1.90 stops and 1.40 stops are reported for the 5× fewer-photon condition, while 2.21 stops and 1.23 stops are reported for the 10× fewer-photon condition.
- C. Estimation error across the dataset: At low summed-frame counts, estimates remain responsive but evidence-limited, especially under reduced light.The evaluated conditions include the original light level, 5× fewer photons, and 10× fewer photons.
- C. Estimation error across the dataset: Increasing Nsum stabilizes estimates, but overly long aggregation introduces blur in regions with scene or camera motion.The optimal summed-frame count is bounded by temporal scene stability.
- C. Estimation error across the dataset: 0.69 stops blur RMSE and 2.02 stops noise RMSE are reported for one evaluated light-level and summed-frame condition.
- C. Estimation error across the dataset: Motion-aware aggregation compares optimal per-pixel integration windows with an optimal global-shutter baseline using signal-to-noise ratio.The analytical upper bound is expressed as a function of expected photon count per frame and integration window.