Source-linked AI summary
Human action recognition with a large-scale brain-inspired photonic computer
Piotr Antonik, Nicolas Marsal, Daniel Brunner, Damien Rontani
TL;DR
Video-based human action recognition is difficult, while deep-learning solutions can demand large datasets and energy-intensive specialized hardware. The paper develops a scalable optoelectronic reservoir computer for this task and achieves accuracy comparable to state-of-the-art approaches, using a flexible architecture with substantial scaling potential.
Problem
Video-based human action recognition is challenging, and deep-learning approaches require large datasets, complex tuning, and energy-consuming specialized hardware.
Method
The paper uses an optoelectronic reservoir computer with parallel optical processing to classify HOG features from six KTH human actions.
Results
92% classification accuracy on KTH is comparable to state-of-the-art rates of 90.7%−95.6% achieved by more complex digital architectures.
Takeaways & Limitations
The system demonstrates a scalable photonic hardware approach for video processing, with 16,384 experimental nodes and potential scaling to hundreds of thousands.
Takeaways & Limitations
The current implementation uses Matlab for processing, limiting speed to up to 7 frames per second.
Abstract
from arXiv · showhide
The recognition of human actions in video streams is a challenging task in computer vision, with cardinal applications in e.g. brain-computer interface and surveillance. Deep learning has shown remarkable results recently, but can be found hard to use in practice, as its training requires large datasets and special purpose, energy-consuming hardware. In this work, we propose a scalable photonic neuro-inspired architecture based on the reservoir computing paradigm, capable of recognising video-based human actions with state-of-the-art accuracy. Our experimental optical setup comprises off-the-shelf components, and implements a large parallel recurrent neural network that is easy to train and can be scaled up to hundreds of thousands of nodes. This work paves the way towards simply reconfigurable and energy-efficient photonic information processing systems for real-time video processing.
1 Introduction
The paper targets the practical challenges of video-based human action recognition by replacing demanding deep-learning hardware and training with a scalable photonic reservoir computer. It evaluates this approach on the KTH action dataset and reports accuracy comparable to state-of-the-art systems.
- 1 Introduction: Video-based human action recognition remains challenging because of clutter, occlusion, viewpoint and scale changes, lighting, and appearance variation.
- 1 Introduction: Deep-learning approaches can require very large datasets, difficult hyperparameter tuning, energy-consuming training, and dedicated high-end hardware.
- 1 Introduction: Reservoir computing processes time series with a random recurrent network while training only a linear output layer, simplifying training and reducing dataset demands.
- 1 Introduction: The proposed optoelectronic reservoir computer uses parallel optical processing and can accommodate 16,384 nodes, with a physical limit of 262,144 neurons.The input, output, and recurrent connections are digitally implemented in this work.
- 1 Introduction: The system classifies six KTH motions from HOG features extracted from video frames recorded across 25 subjects.
2 Results
The optical reservoir computer combines HOG/PCA preprocessing with frame-wise recurrent classification and majority voting to recognise six human actions. Across simulations and experiments, performance improves with reservoir size and approaches digital state-of-the-art while retaining flexible optical processing.
- 2.1 Classification of human action with a reservoir computer: The system extracts HOG features, reduces their dimensionality with PCA, classifies frames using six binary outputs, and assigns each video the majority frame class.Winner-takes-all selects the frame class with the maximum output before sequence-level majority voting.
- 2.2 Photonic reservoir computer: The optical setup implements reservoir dynamics with an SLM, camera, nonlinear optical processing, randomly generated input masks, and recurrent interconnection weights.Camera images are cropped and averaged over macro-pixels to obtain reservoir states used for subsequent updates and output computation.
- 2.2 Photonic reservoir computer: Processing reaches 2 video frames per second for N = 16,384 and up to 7 frames per second for N = 1,024 because Matlab limits the current setup.The authors state that DSP, FPGA, or fully parallel optical matrix products could alleviate this limitation.
- 2.3 Reservoir size and classification performance: Experimental performance rises steeply through N = 4,096 nodes, reaching an average score of 548, while N = 16,384 produces only a 1.3% experimental drop.The larger-reservoir downturn is attributed to tilt and misalignment becoming more noticeable as macro-pixels shrink.
- 2.3 Reservoir size and classification performance: The photonic system is 4.7% below the best s1 result but processes faster than SVM by a factor of ten and trains faster than deep approaches.Its training time is comparable to an SVM with hierarchical compound features.
3 Discussion & conclusion
The paper presents a scalable photonic reservoir computer for human-action recognition, combining simple training with substantial parallel hardware capacity. Its 16,384-node implementation achieves performance comparable to state-of-the-art approaches and could support parallel processing of multiple video feeds.
- 16,384 physical nodes demonstrate a scalable photonic architecture with potential expansion to hundreds of thousands of nodes.The architecture could allocate separate SLM regions to independent reservoir computers for parallel video streams.
- The photonic reservoir computer achieves KTH-dataset performance comparable to state-of-the-art deep approaches and superior to gradient-optimised LSTM networks.
- The system simplifies training by using a shallow reservoir computer while leveraging optical parallelism for hardware-based video processing.
4 Methods
The methods combine a discrete-time reservoir computer with an optoelectronic photonic implementation, HOG frame features, and grid-searched hyperparameters for KTH action classification.
- 4.1 Basic principles of reservoir computing: The reservoir produces six action outputs from recurrent states using a linear readout trained to minimise normalized mean square error.The readout weights are trained offline with linear regression, including ridge regression.
- 4.2 Physical modeling of the photonic reservoir computer: The photonic neuron state is camera-detected 10-bit optical intensity, updated from masked input and adjacency-matrix transformations before nonlinear optical conversion.The SLM phase is converted into intensity through the setup’s polarization configuration and a sin^2 nonlinearity.
- 4.2 Physical modeling of the photonic reservoir computer: The system was tested with both intensity- and phase-based state models, with trainable readout matrices producing one output for each of six actions.The alternative model uses the SLM macro-pixel phase shift as the neuron state.
- 4.3 Hyperparameters: The four hyperparameters are input gain, feedback gain, interconnection gain, and interconnection density, selected by exhaustive grid search across specified intervals.The optimal values were recorded for selected reservoir sizes in Table 3.
- 4.3 Hyperparameters: Input and feedback gains strongly affect accuracy, whereas interconnection-matrix characteristics have a minor role and substantially different matrices can yield comparable scores.The comparison concerns matrix density and off-diagonal amplitude.
- 4.4 The KTH dataset: The KTH setup uses six actions from 25 subjects with four repetitions, while HOG converts each 160×120 grayscale frame into 9576 features.The dataset contains 600 sequences, and HOG uses 8×8 cells with 2×2 blocks.
5 Author contributions statement
The study was designed and managed by D.B., N.M., and D.R., with distinct contributions to experimentation, simulations, and manuscript preparation.
- 5 Author contributions statement: D.B., N.M., and D.R. designed and managed the study, while P.A., N.M., and D.R. built the experimental setup.P.A. conducted the numerical simulations and experimental campaigns; P.A., N.M., and D.R. prepared the manuscript.