Source-linked AI summary
Deep Fully-Connected Networks for Video Compressive Sensing
Michael Iliadis, Leonidas Spinoulas, Aggelos K. Katsaggelos
TL;DR
Video compressive sensing seeks high-speed video reconstruction despite camera frame-rate limits and the slow recovery of existing methods. The paper learns a measurement-to-video mapping, first linearly and then with deep fully connected networks, and reports improved reconstruction quality with recovery in a few seconds. Its conclusions are based on simulations and point to dataset size, depth, alternative architectures, and real-camera sequences as areas for further investigation.
Problem
Video compressive sensing is limited by camera frame rates and by reconstruction algorithms that can make recovery prohibitively slow.
Method
The paper learns a direct mapping from temporal compressive-sensing measurements to video frames, extending linear regression to deeper fully connected networks.
Results
FC7-10M achieves 32.93 dB average PSNR in about 12 seconds, while FC7-10M+MMLE reaches 33.58 dB but requires almost two hours.
Takeaways & Limitations
The framework reduces reconstruction time to a few seconds while improving reconstruction quality compared with existing approaches on simulated video sequences.
Takeaways & Limitations
The reported quantitative comparisons use only the first 32 frames of each sequence because previous methods have extremely long reconstruction times.
Abstract
from arXiv · showhide
In this work we present a deep learning framework for video compressive sensing. The proposed formulation enables recovery of video frames in a few seconds at significantly improved reconstruction quality compared to previous approaches. Our investigation starts by learning a linear mapping between video sequences and corresponding measured frames which turns out to provide promising results. We then extend the linear formulation to deep fully-connected networks and explore the performance gains using deeper architectures. Our analysis is always driven by the applicability of the proposed framework on existing compressive video architectures. Extensive simulations on several video sequences document the superiority of our approach both quantitatively and qualitatively. Finally, our analysis offers insights into understanding how dataset sizes and number of layers affect reconstruction performance while raising a few points for future investigation. Code is available at Github: https://github.com/miliadis/DeepVideoCS
1 Introduction
Temporal video compressive sensing addresses limited camera frame rates by reconstructing high-speed video from multiplexed measurements, but existing reconstruction methods are often too slow. This work applies deep learning, beginning with linear regression and extending to fully connected networks, to improve reconstruction quality and reduce recovery time.
- High-speed imaging remains constrained by fundamental limits on achievable camera frame rates despite cheaper high-speed hardware.
- Temporal compressive sensing multiplexes video information across time, producing low-frame-rate coded measurements that can be decoded into high-speed sequences.
- Existing temporal compressive sensing reconstruction algorithms can be computationally intensive enough to make video recovery impractical for commercial cameras.
- The proposed approach learns a direct mapping from temporal compressive sensing measurements to video frames using fully connected neural networks and repeated-pattern measurement masks.
- A linear regression model reconstructs video frames adequately at minimal computational cost and can initialize other video compressive sensing algorithms.
- Deeper architectures improve reconstruction quality and computational cost relative to previous methods while targeting recovery in a few seconds.
2 Motivation and Related Work
Deep learning has achieved strong results in restoration tasks, motivating its application to video compressive sensing. The paper focuses on the less straightforward problem of recovering a higher-dimensional 3D video volume from lower-dimensional 2D measurements.
- Deep learning has achieved state-of-the-art performance in image recognition, object detection, and several image restoration problems.
- Fully connected networks, stacked denoising auto-encoders, convolutional networks, and recurrent networks represent major architectures used for restoration tasks.
- Estimating Nf unknowns from Mf inputs where Mf ≪ Nf is not straightforward because common learning architectures usually preserve or reduce dimensionality.
- Prior deep-learning compressive-sensing studies addressed spatial compressive sensing for still images, whereas this work targets temporal video compressive sensing.
- The proposed task differs from 2D image restoration because it reconstructs a 3D volume from 2D measurements.
3 Deep Networks for Compressed Video
The paper first learns a linear block-wise mapping from compressed measurements to video blocks, then extends it to fully connected deep networks using a repeated, hardware-realizable measurement pattern. The proposed MLPs reconstruct 3D video blocks from 2D measurements, with experiments examining architecture depth, training data, and reconstruction quality.
- 3.1 Linear mapping: Training learns a linear mapping Wp that reconstructs video blocks from measurements yi produced by the same block measurement matrix Φp.The mapping is estimated from paired training blocks and measurements, with Wp having size Np × Mp.
- 3.1 Linear mapping: With sufficiently large training data and at least one nonzero measurement per spatial location, the ill-posed linear approach provides surprisingly good reconstruction.The condition ensures that each spatial location is sampled at least once over time.
- 3.1 Linear mapping: The linear experiments evaluate average PSNR and SSIM on 14 unrelated test videos using two random binary measurement matrices with varying nonzero percentages.The setup uses 8 × 8 × 16 blocks, simulating 16-frame reconstruction from one captured frame, with N = 10^6 training blocks.
- 3.1 Linear mapping: A repeated three-dimensional building block constructs a full measurement matrix so every block uses the same pattern and overlapping blocks can be reconstructed.The selected building block is 4 × 4 × 16 with 50% nonzero elements; 8 × 8 × 16 reconstruction blocks overlap spatially by 4 × 4.
- 3.3 Multi-layer Network Architecture: The nonlinear extension uses end-to-end MLPs that map measured frame patches to video blocks through multiple fully connected hidden layers.The first fully connected layer expands compressed 2D measurements into a 3D signal, while later convolutions are avoided because small blocks make them ineffective and larger blocks increase complexity.
- 3.3 Multi-layer Network Architecture: The networks use ReLU hidden layers, consider four- and seven-layer architectures, and train all weights and biases by minimizing mean squared error.MSE is selected because it is directly related to the PSNR objective.
4 Experiments
The experiments evaluate fully connected video compressive-sensing reconstructions against established methods using quantitative, qualitative, noise-robustness, runtime, and architecture-size analyses. The proposed networks achieve stronger reconstruction quality and much faster inference, while dataset size and depth influence performance under different conditions.
- Experimental setup: The study compares proposed deep architectures with state-of-the-art methods quantitatively and qualitatively using PSNR and SSIM, including noiseless and noisy measurements.Experiments use 400 high-definition training videos and up to 10 million extracted video blocks, with standardized optimization settings.
- Comparison with previous methods: FC7-10M and FC7-10M+MMLE achieve the highest PSNR and SSIM for all video sequences.FC7-10M improves average PSNR over GMM-1 by 2.15 dB, while FC7-10M+MMLE improves over GMM-1+MMLE by 1.67 dB.
- Runtime: 12 seconds versus almost two hours: FC7-10M reconstructs 16 frames far faster than FC7-10M+MMLE while retaining 32.93 dB average PSNR.The paper cautions that runtime comparisons are not trivial because implementations differ, although the proposed MLP uses feed-forward matrix-vector multiplications.
- Reconstruction results: The proposed MLPs favorably recover motion, while deeper models produce sharper edges and more high-frequency scene details than previous algorithms.These qualitative comparisons are shown for selected frames from two video sequences.
- Reconstruction results: Because previous methods require extremely long reconstruction times, the main quantitative comparisons cover only the first 32 frames of each video sequence.The paper separately compares PSNR across all frames of six sequences using FC7-10M and GMM-4.
- Reconstruction with noise: Noisy-data training makes FC7N-10M more robust than GMM-1 across all tested measurement-noise levels.FC7-10M performs well at low noise but reaches performance similar to GMM-1 at more severe noise levels.
- Number of layers and dataset size: Performance consistently improves with more training samples, whereas increasing depth from four to seven layers yields little gain on small datasets.On the 10M dataset, average PSNR is 32.66 dB for FC4-10M and 32.91 dB for FC7-10M.
5 Conclusions
The work introduces a deep learning architecture for temporal video compressive sensing that improves performance over existing algorithms while reducing reconstruction time to a few seconds.
- The proposed architecture achieves superior performance compared to existing algorithms while reducing reconstruction time to a few seconds.
- The framework is designed for applicability to existing compressive camera architectures, suggesting potential commercial viability.
- Future directions include recurrent architectures, deeper networks, and evaluation on real video captured by temporal compressive sensing cameras.