PaperlayerView exact revision on arXiv

Source-linked AI summary

1000x Faster Camera and Machine Vision with Ordinary Devices

Tiejun Huang, Yajing Zheng, Zhaofei Yu, Rui Chen, Yuan Li, Ruiqin Xiong, Lei Ma, Junwei Zhao, Siwei Dong, Lin Zhu, Jianing Li, Shanshan Jia, Yihua Fu, Boxin Shi, Si Wu, Yonghong Tian

arXiv:2201.09302v1cs.CV

TL;DR

Frame-based cameras miss changes in light during exposures and between frames, limiting capture of rapidly changing scenes. The paper introduces vidar, a threshold-triggered spike representation, and an SNN vision system. The camera and vision system are reported to operate 1,000 times faster than conventional cameras and human vision, respectively.

  • Problem

    Frame-based images and videos miss light changes during exposures and between frames, while consumer-level cameras sample at only tens of hertz.

    Source: §1 · p. 2

  • Method

    Vidar represents threshold-triggered photon accumulation as spatially arranged spike sequences, which an SNN processes for machine vision.

    Source: §1 · p. 4, §3.1 · pp. 15–16

  • Results

    The vidar camera samples at 40,000 Hz, reported as 1,000 times faster than human vision and conventional cameras, while SNN-based detection and tracking are reported 1,000 times faster than human vision.

    Source: §1 · p. 4, §3.1 · p. 17

  • Takeaways & Limitations

    The authors demonstrate vidar-camera full-time imaging for judging a ball's landing location and describe consumer-device prospects including mobile phones and cameras.

    Source: §3.5 · p. 30, §3.6 · p. 34

    In a readout cycle, the chip processes only one spike signal, even if two or more are triggered.

    Source: §3.2 · p. 21

Abstract

from arXiv · show

In digital cameras, we find a major limitation: the image and video form inherited from a film camera obstructs it from capturing the rapidly changing photonic world. Here, we present vidar, a bit sequence array where each bit represents whether the accumulation of photons has reached a threshold, to record and reconstruct the scene radiance at any moment. By employing only consumer-level CMOS sensors and integrated circuits, we have developed a vidar camera that is 1,000x faster than conventional cameras. By treating vidar as spike trains in biological vision, we have further developed a spiking neural network-based machine vision system that combines the speed of the machine and the mechanism of biological vision, achieving high-speed object detection and tracking 1,000x faster than human vision. We demonstrate the utility of the vidar camera and the super vision system in an assistant referee and target pointing system. Our study is expected to fundamentally revolutionize the image and video concepts and related industries, including photography, movies, and visual media, and to unseal a new spiking neural network-enabled speed-free machine vision era.

1. Introduction

Conventional image and video formats miss light changes during exposures and between frames, limiting consumer cameras on fast scenes. Vidar encodes temporal light changes as spike sequences, enabling fast imaging and SNN-based vision with ordinary devices.

1. Introduction

  • Frame-based images lose light changes during exposures, while videos miss dynamics between exposures; consumer cameras operate at only tens of hertz.High-speed cameras reach thousands or tens of thousands of hertz but require expensive specialized sensors and shutters.Source: §1 · p. 2
  • Vidar replaces frame-based representation with spike sequences that retain the timing of optical flow and support scene-radiance reconstruction at any moment.The paper calls this capability full-time imaging.Source: §1 · pp. 2–3
  • Using consumer-grade CMOS sensors and integrated circuits, the VidarOne chip generates a spike when accumulated light intensity crosses a threshold and samples at 40,000 Hz.The authors report this as 1,000 times faster than human vision and conventional cameras.Source: §1 · pp. 3, 5
  • An SNN processes vidar spike streams in real time, achieving moving-object detection and tracking 1,000 times faster than human vision.The system processes 40,000-Hz streams on ordinary CPUs; conventional ANN methods first convert streams into frames and process them frame by frame.Source: §1 · p. 4

2. Methods

The methods combine vidar-based image reconstruction with SNN mechanisms for filtering, detection, tracking, prediction, and recognition. The system is evaluated on reconstructed-image quality and object-vision tasks, including high-speed targets.

2. Methods

  • TFW estimates pixel values by counting spikes in a moving time window, while TFI estimates them from an interspike interval under a short-period constant-radiance assumption.The reconstruction methods outperform a conventional camera on both 2-D entropy and standard deviation.Source: §2 · pp. 5–6, Table 1 · p. 7
  • The dynamic connection gate uses short-term plasticity: PSP settles for fixed-frequency input and fluctuates around a stable value for varying-frequency input, filtering static-background spikes while retaining moving-object spikes.The figure illustrates the fixed-frequency case at 30 Hz.Source: §2 · pp. 7–8, Figure 1 · p. 8
  • Detection-layer LIF neurons accumulate input and fire at threshold; connected firing areas identify moving objects, which are tracked using tracking-by-detection.The authors report good detection and tracking performance with low power, evaluating detection with DSP and tracking with MOTA, FP, FN, and IDS.Source: §2 · pp. 8–9, Table 2 · p. 9
  • A CANN with negative feedback is used for anticipatory tracking of fast-moving objects, with visual input supplied directly by the vidar camera.The model builds on prior work reporting approximately constant leading time for anticipatory tracking.Source: §2 · p. 9
  • The recognition SNN uses BP-STDP-trained synaptic weights and groups multiple output neurons to represent each category, training with real and simulated vidar spike data.Target-group neurons undergo STDP updates, while misfired neurons in non-target groups undergo anti-STDP.Source: §2 · pp. 9–10
  • 40 m/s: the vidar camera and super vision system detect, track, and recognize a moving object within 1 m in real time.The paper estimates the fan-mounted character speed from a 2,400-rpm rotation and a 0.12-m radius.Source: §2 · p. 10
  • VHSSD contains static-camera recordings of moving targets and moving-camera recordings of natural scenes; Vidar-Sim generates spike streams for arbitrary camera and object motion.VidarPlayer plays real and simulated spatial-temporal spike streams and supports multiple resolutions.Source: §2 · pp. 11–12

3. Results

The results describe vidar’s continuous, threshold-triggered spike-stream approach to preserving temporal changes and reconstructing scenes, alongside the VidarOne camera and an SNN-based super vision system. They demonstrate high-speed object detection, tracking, and recognition, and outline potential applications and camera advantages.

3.1 Vidar: a new and more natural visual form

  • Vidar continuously samples photons and encodes threshold-triggered spikes as spatially arranged bit streams, retaining temporal changes that frame-based video loses.Each unit emits a spike when accumulated intensity crosses a threshold; the spike and time to generate it form a vit.Source: §3.1 · p. 16
  • 40,000 Hz sampling on the first chip is reported as 1,000 times faster than human vision and traditional cameras.Vidar can reach 10 million hertz sampling, and its spike streams support full-time imaging by estimating scene radiance at arbitrary moments.Source: §3.1 · pp. 17–18

3.2 VidarOne chip and vidar camera system

  • The 400×250-pixel VidarOne array uses a 100 ns row scan, yielding 25 μs time resolution for spike streams across its 250 rows.The chip uses an asynchronous pixel-trigger architecture with rolling-shutter response detection and parallel readout.Source: §3.2 · pp. 18, 22

3.3 Visual texture reconstruction with the vidar camera

  • TFW reconstructs texture from spike counts for stationary scenes, whereas TFI estimates radiance from interspike intervals and better captures high-speed motion.TFI updates motion nearly synchronously; the reported reconstruction methods outperform conventional cameras quantitatively, though the supplied passage gives no metric values.Source: §3.3 · pp. 23–24, Figure 4 · p. 25

3.4 Super vision system with SNNs

  • The SNN super vision system performs high-speed object detection, tracking, prediction, and recognition at a reported 1,000 times the speed of human vision.It combines a short-term-plasticity dynamic connection gate, locally connected detection and tracking SNN, CANN trajectory prediction, and fully connected SNN recognition.Source: §3.4 · pp. 26–28
  • The SNN super vision system combines a dynamic connection gate that filters fixed-frequency spikes, a CANN for trajectory prediction, and SNNs for object detection, tracking, and recognition.The recognition network is three-layer fully connected; the system framework also includes prediction.Source: Figure 5 · p. 29, §3.4 · p. 29

3.5 Demonstration of the utility of the vidar camera and the super vision system

  • For auxiliary refereeing, vidar captures the entire landing process of an approximately 100 km/h table tennis ball, allowing its landing location to be judged against the boundary.The passage contrasts this full-time record with systems that calculate the landing moment from the ball's trajectory.Source: §3.5 · pp. 30–31
  • On a fan rotating at approximately 2400 rpm, the system detects and tracks three character targets, recognizes a selected character, predicts its trajectory, and guides a laser to hit it.The demonstration uses characters P, K, and U; the recognition output neuron for the correct category produces the most spikes.Source: §3.5 · pp. 31–32
  • 30 m/s fan motion is detected, tracked, and recognized in real time within 0.75 m; the authors project aircraft at the speed of sound within 10 m and Mach 100 objects within 1 km.The aircraft and Mach 100 figures are projections based on the central perspective principle.Source: §3.5 · p. 33

3.6 Application prospects

  • Unlike expensive high-speed cameras requiring specialized sensors and shutters, vidar is built from conventional CCDs using regular semiconductor manufacturing and may suit everyday devices such as phones and cameras.The paper attributes traditional cameras' difficulty with fast motion to information loss between frames.Source: §3.6 · pp. 33–34
  • Vidar reconstructs scene radiance at any moment by generating spikes when accumulated intensity crosses a threshold, unlike DVS brightness-change events; the authors anticipate surveillance and SNN-based visual applications.Suggested surveillance uses include dynamic face, fingerprint, and palm-print recognition; the authors also anticipate object detection, recognition, and tracking at electrical speeds.Source: §3.6 · pp. 34–35

4. Conclusion

The conclusion presents vidar as a route to replace traditional video cameras, especially for high-speed scenes. It frames vidar’s spike stream as a natural input for SNNs in machine vision.

4. Conclusion

  • Vidar is proposed as a replacement for traditional video cameras in almost all fields, especially high-speed scenes.The conclusion links this proposal to realizing optoelectronic technology's potential, which it says has been suppressed for decades.Source: §4 · p. 35
  • Vidar’s spike stream characterizes optical changes across space and time and is a natural input for SNNs.Source: §4 · p. 35
  • The conclusion casts vidar as a new generation of the eye of machine vision in the era of artificial intelligence.Source: §4 · p. 35
Loading 2201.09302v1…