Streaming quanta sensors for online, high-performance imaging and vision

I. INTRODUCTION

• QIS can produce read-noise-free binary frames at up to 100 kFPS, but a 1-megapixel sensor at that rate generates 100 Gbps and requires costly processing.The paper describes the resulting bandwidth, power, memory, and computation demands as obstacles to real-time deployment.Source: §I · p. 1
• The streaming representation retains scene information at multiple temporal scales, and a feed-forward network reconstructs intensity frames with quality comparable to resource-intensive approaches.The architecture combines an extreme multi-exposure stack with a modified U-Net containing ResNet blocks.Source: §I · p. 1, Figure 1 · p. 2
• The representation reduces raw-bitplane bandwidth by about two orders of magnitude with 16 floating-point operations per pixel per binary-frame update.The feed-forward reconstruction network runs at tens of frames per second on a desktop GPU and is reported to be 4–5 orders of magnitude faster than existing QIS reconstruction methods.Source: §I · p. 2
• The paper provides a semi-realistic dataset combining real-camera and 3D-simulated data for supervised learning on dynamic QIS problems.It includes diverse object categories and motion statistics and is described as generalizing across light levels.Source: §I · p. 2
• The authors apply the representation and algorithms to face detection, object detection, human pose estimation, and tracking, reporting near-real-time performance.Source: §I · p. 2

A. Image processing and vision with QIS

• Earlier QIS processing demonstrated low-light and high-dynamic-range imaging, while motion-compensation methods faced difficulty with complex motion and occlusion.Quanta Burst Photography handles more realistic scenes but requires all raw binary data, can take minutes or more, and relies on optical-flow guidance that may fail over longer horizons.Source: A. Image processing and vision with QIS · p. 2

B. QIS and SPAD datasets for supervised learning

• Paired binary-frame and ground-truth intensity datasets are difficult to capture at QIS speeds and in low light.Commercial high-speed sensors with full bit-depth can have read noise up to 10e- rms, yielding extremely low SNR under moderate or low light.Source: B. QIS and SPAD datasets for… · p. 3
• Existing synthesized dynamic datasets use simplified motion or interpolated video, with some failing to represent self-occlusion and depth-driven spatially varying motion.The passage describes these limitations as motivating the paper's data-generation approach.Source: B. QIS and SPAD datasets for… · p. 3

C. Vision with QIS under sparse photon arrivals

• QIS has supported low-light vision with fewer than one photon per pixel for classification, detection, and depth prediction; dynamic-scene work has also applied off-the-shelf vision networks after image reconstruction.Source: C. Vision with QIS under sparse photon… · p. 3

D. Streaming algorithms and perception

• Streaming perception seeks to keep visual processing aligned with the world's latest state, motivating the paper's low-latency QIS representation.The cited streaming-algorithm account describes real-time, one-pass handling of fast data streams.Source: D. Streaming algorithms and perception · p. 3

E. Multi-exposure architectures and algorithms

• Multi-bucket sensors capture different exposures simultaneously, whereas cited fusion methods are typically capture-specific and assume short exposures have enough photons for direct fusion.That high-photon assumption does not hold for QIS, whose short exposures contain heavy shot noise.Source: E. Multi-exposure architectures and… · p. 3

III. QUANTA STREAMING REPRESENTATION

• The proposed representation is designed to summarize multiple timescales, update every quanta frame with few per-pixel operations, and reduce memory by about 100× versus raw quanta data.Its virtual exposure range is intended to balance noise and motion blur across object velocities and applications.Source: §III · p. 3

A. Streaming multi-exposure mean representation

• The representation applies exponentially weighted means to the binary stream, using faster decay for noisy, low-blur motion information and slower decay for higher-SNR information that may blur motion.Each mean can be updated with one multiplication and one addition per pixel per step; storing the means as 8-bit images keeps memory to a small number of conventional images.Source: A. Streaming multi-exposure mean… · pp. 3–4
• Across the exposure stack, short exposures provide contours for fast-moving objects while longer exposures reveal motion trajectories, intensity detail, and low-motion background structure.Source: Figure 2 · p. 4
• The chosen exposure windows range from 16 to 4096 frames, spanning fast through normal human motion while using 8× conventional-sensor memory and about 100× less than raw quanta data.The passage notes that multiple exponents require proportionally more memory and that the minimum effective integration period is about 16 binary frames.Source: A. Streaming multi-exposure mean… · p. 4

B. Advantages of the compact streaming representation

• The representation updates every 10 µs with near-zero lag, reducing sensing latency relative to fixed-duration frame-based capture.The passage contrasts conventional 30–60 FPS sensing latency of about 15–30 ms, which can approach one second for long or multiple exposures.Source: B. Advantages of the compact streaming… · pp. 4–5
• Downstream algorithms can access the current representation at application-dependent, asynchronous rates independent of the sensor sampling rate.The paper gives face detection at about 10 FPS and robotic vision at about 1000 FPS as an example of simultaneous use.Source: B. Advantages of the compact streaming… · p. 5
• The authors attribute the suitability of rapid streaming updates to SPAD-QIS's lack of read noise, which allows fine-scale intensity information to be captured with shot noise alone.Source: B. Advantages of the compact streaming… · p. 5

C. Visual Analysis of the Streaming Representation

• Long and short exposures provide information at different temporal scales, allowing the representation to resolve objects across a large range of velocities.Long exposures support high dynamic range and noise resolution but suit slow-moving objects; short exposures capture semantic and location information for fast-moving objects.Source: C. Visual Analysis of the Streaming… · p. 5
• High-to-medium exposures capture low-speed background objects, while short-to-medium exposures resolve motion from fast-moving objects.The exposure set combines photon integration and motion information across time scales and flux intensities.Source: C. Visual Analysis of the Streaming… · p. 5

IV. LEARNING-BASED IMAGE/VIDEO RECONSTRUCTION

• A U-Net architecture is used to reconstruct high-quality images from the multi-exposure set.The section also discusses architecture and loss design, followed by systematic creation of synthetic training datasets.Source: §IV · p. 5

A. Image reconstruction neural network

• The feed-forward U-Net takes an 8-channel extreme multi-exposure set and outputs a single-channel grayscale image aligned with the latest time instant.The target is the state represented in the shortest exposure; the architecture uses residual blocks, pixel-shuffle downsampling, and interpolation-based upsampling.Source: A. Image reconstruction neural network · p. 5
• The low-light loss combines L1 image error with L1 error on image gradients, weighted by σ = 0.1.The ground-truth frame is temporally aligned with the latest binary frame, and gradient terms preserve high-frequency components and fine details.Source: A. Image reconstruction neural network · p. 5
• The network minimizes loss on tone-mapped intensity to learn across a wide range of photon fluxes.The paper uses a differentiable µ-law function for the tone mapping.Source: A. Image reconstruction neural network · p. 5
• The µ-law compression rate is µ = 103, and ReLU with ζ = 10−7 prevents nonpositive arguments to the logarithm.The ReLU and stability parameter are applied to prevent negative or zero log-function arguments.Source: A. Image reconstruction neural network · pp. 5–6
• Figure 3 contrasts conventional-camera read noise with QIS operation and compares latency across multi-exposure architectures.Its caption states that streaming multi-exposures provide the latest information at any time, reducing downstream inefficiencies.Source: Figure 3 · p. 6

B. QIS dynamic dataset

• The dataset combines RIFE-interpolated NFS footage [32] with Kubric-rendered motion, covering diverse velocities, objects, occlusions, and scenarios.NFS footage is captured at 240 FPS and interpolated 16×; the dataset targets motion, light, texture, and sensor-characteristic diversity for QIS training.Source: B. QIS dynamic dataset · p. 6, Figure 4 · p. 7

1) Quanta sensor model and simulator:

• Binary frames are simulated from Poisson photon arrivals plus dark counts, followed by pileup binarization; the model assumes constant PDP and uniform dark counts.Dark counts are calibrated from real sensor measurements, and high-dark-count pixels are masked before the neural network.Source: 1) Quanta sensor model and simulator · pp. 6–7
• At 20 dB, 4096 frames resolve [0.025, 7] and 256 resolve [0.06, 2.5] photons/pixel/frame; experiments extend the flux range to [0.01, 10].The extension beyond the estimated 20 dB range is attributed to neural-network denoising.Source: 1) Quanta sensor model and simulator · p. 7, Figure 5 · p. 7
• HDR augmentation applies a thresholded two-segment transfer to clean frames, sampling λlow from 0.01–0.1 and λhigh from 0.2–10, with a threshold around 0.8.The resulting flux-augmented sequences serve as ground-truth references.Source: 1) Quanta sensor model and simulator · pp. 7–8

• The experiments evaluate reconstruction on both synthetic and real QIS data.Source: §V · p. 8
• QBP is the state-of-the-art reconstruction algorithm used as a comparison baseline.Source: §V · p. 8
• Naive integration of binary frames is also compared across multiple exposure periods.Source: §V · p. 8

A. Training details

• Training uses 20,000 pregenerated samples and a U-Net trained for 200 epochs with ADAM at a 3×10^-4 learning rate.Samples are cropped to 256×256, with random horizontal or vertical flips applied online during training.Source: A. Training details · p. 8

B. Synthetic experiments and results

• Synthetic reconstructions are comparable in quality to QBP, with lightweight representations and online processing capability.Fig. 6 shows Kubric low-light and NFS HDR examples; the reported comparisons use PSNR and SSIM.Source: B. Synthetic experiments and results · p. 8, Figure 6 · p. 8

C. Real data

• Real-data reconstructions are qualitatively comparable to QBP and remain sharp under significant motion and very low light.Evaluation is qualitative because ground-truth intensity data are unavailable.Source: C. Real data · p. 8

D. Runtime comparison

• On real global-motion SPAD-QIS scenes, both QBP and our method produce sharp images and recover details, while 4096-frame naive integration blurs under rapid motion.The examples include indoor toys and night driving; our method uses less bandwidth and computation than QBP.Source: Figure 7 · p. 9
• 12 ms per reconstruction on an A100 versus QBP’s 1.5–2 minutes on an Intel Xeon E3-1220 v2 for 4096 frames; lower-end desktop GPUs still reach tens of FPS.The real-scene comparison also reports sharp reconstructions with reduced bandwidth and computation, while 4096-frame naive integration (≈0.3 s) produces motion blur.Source: D. Runtime comparison · pp. 8–9, Figure 7 · p. 9

E. Characterizations using synthetic data

• Across light-level tests, our approach gives the overall best PSNR and SSIM; at very low light, its quality is comparable to QBP with BM3D, with both heavily limited at 100 photons per second.The tests fix speed at 1000 pixels/second and vary maximum light levels from 100 to 10k photons per second.Source: E. Characterizations using synthetic… · p. 9
• As motion speed increases, our method performs similarly to QBP but degrades; beyond 1000 pixels/s, QBP cannot properly align to the latest intensity frame, possibly because optical flow fails near the sequence end.The comparison uses a fixed photon rate averaging about 200 photons/second/pixel.Source: E. Characterizations using synthetic… · p. 9

VI. VISION APPLICATIONS

• The vision pipeline applies off-the-shelf inference models to reconstructed images at tens of FPS, with later work proposed on end-to-end training.The real-data examples also include a swinging pendulum against a dark background and a walking person whose face is reconstructed.Source: §VI · p. 9, Figure 8 · p. 10

A. Object detection (20 FPS)

• Fig. 9 reports comparable or better PSNR/SSIM than QBP across lighting and motion configurations.The performance plots summarize 250 samples per light/motion point.Source: Figure 9 · p. 10
• 5 FPS for MaskRCNN versus 20 FPS for YOLOv5; MaskRCNN is more robust, while YOLOv5 gives less stable predictions.The real-data detection examples include low-light scenes from dark rooms to night streets, with fast subject or camera motion; the caption reports good detection results.Source: A. Object detection (20 FPS) · p. 11, Figure 10 · p. 11

B. Human-interaction tasks: face and pose detection

• 20 FPS for both face detection and pose estimation, with good-quality results on real QIS captures.The examples include low-light scenes with fast subject motion, where the tasks can run in real time.Source: B. Human-interaction tasks: face and… · p. 11, Figure 11 · p. 11

A. Limitations

• QBP may preserve fine details more faithfully; this method is not targeted at extremely high flux, and supervised-network artifacts and training-data distribution can affect performance.The paper says QBP may be more reliable for fine-detail tasks such as face recognition or barcode scanning, while this method better suits bandwidth- and real-time-critical settings.Source: A. Limitations · p. 11

B. End-to-end vision

• Future work could train directly on the multi-exposure representation and pair streamed representations with labels generated from interpolated videos or simulated data.The paper names NFS, Kubric, and PointOdyssey as possible data sources or generation frameworks.Source: B. End-to-end vision · p. 11

C. Tracking

• Siamese-RPN tracks a running person in the reconstructions, with the first-frame human detection as the template and later positions shown by red boxes.The night-time example is described as reliable tracking despite the challenging scene.Source: C. Tracking · p. 11, Figure 12 · p. 12

C. Other streaming representations and inference architectures

• Other proposed directions include higher-order statistics, projections, coding, differential approaches, multi-spatial resolutions, and color acquisition using a Bayer filter.Source: C. Other streaming representations and… · p. 12