Survey on Deep Neural Networks in Speech and Vision Systems

Mahbubul Alam; Manar D. Samad; Lasitha Vidyaratne; Alexander Glandon; Khan M. Iftekharuddin

Source-linked AI summary

Survey on Deep Neural Networks in Speech and Vision Systems

Mahbubul Alam, Manar D. Samad, Lasitha Vidyaratne, Alexander Glandon, Khan M. Iftekharuddin

arXiv:1908.07656v2Aug 16, 2019cs.CVcs.LGcs.NEcs.SDeess.ASeess.SPstat.ML

TL;DR

The survey addresses how deep neural architectures and systems advance vision and speech applications while remaining difficult to deploy on resource-constrained platforms. It reviews architectures, applications, hardware adaptations, and limitations, concluding that deep learning has enabled broad intelligent-system progress but still faces substantial mobile and embedded deployment challenges.

Problem
Vision and speech systems must overcome limitations of hand-engineered processing and the computation, memory, power, and latency constraints of mobile and embedded hardware.
Method
The paper systematically reviews deep neural architectures, algorithms, applications, hardware implementations, and approaches for compact and efficient deployment.
Results
The survey finds CNNs dominant in vision, RNNs leading speech and emerging NLP applications, and compression and quantization reducing mobile memory and computational requirements.
Takeaways & Limitations
Deep learning has pioneered new intelligent vision and speech applications, while efficient architectures remain important for standalone operation on mobile and resource-constrained devices.
Takeaways & Limitations
Deep models still require substantial training data and heavy computation, memory, and power resources that restrict practical deployment.

Abstract

from arXiv · show

This survey presents a review of state-of-the-art deep neural network architectures, algorithms, and systems in vision and speech applications. Recent advances in deep artificial neural network algorithms and architectures have spurred rapid innovation and development of intelligent vision and speech systems. With availability of vast amounts of sensor data and cloud computing for processing and training of deep neural networks, and with increased sophistication in mobile and embedded technology, the next-generation intelligent systems are poised to revolutionize personal and commercial computing. This survey begins by providing background and evolution of some of the most successful deep learning models for intelligent vision and speech systems to date. An overview of large-scale industrial research and development efforts is provided to emphasize future trends and prospects of intelligent vision and speech systems. Robust and efficient intelligent systems demand low-latency and high fidelity in resource-constrained hardware platforms such as mobile devices, robots, and automobiles. Therefore, this survey also provides a summary of key challenges and recent successes in running deep neural networks on hardware-restricted platforms, i.e. within limited memory, battery life, and processing capabilities. Finally, emerging applications of vision and speech across disciplines such as affective computing, intelligent transportation, and precision medicine are discussed. To our knowledge, this paper provides one of the most comprehensive surveys on the latest developments in intelligent vision and speech applications from the perspectives of both software and hardware systems. Many of these emerging technologies using deep neural networks show tremendous promise to revolutionize research and development for future vision and speech systems.

1. INTRODUCTION

The survey motivates deep learning for vision and speech by identifying limits of hand-engineered features and reviewing hierarchical neural architectures that learn representations from raw data. It also frames deployment challenges arising from computation, memory, power, and latency constraints.

Hand-engineered features depend on application-specific human interpretation and require separate feature extraction before machine learning.
Deep learning emerged from advances in trainable data, learning algorithms, and computational resources, enabling applications across vision, speech, biomedical imaging, and online analysis.
The survey covers architectures, applications, hardware-restricted implementations, and future intelligent-system prospects.
Deep neural networks arrange nonlinear modules hierarchically so models learn meaningful features at different abstraction levels.
Convolutional neural networks: CNNs learn hierarchical image patterns through 2D convolutions using local connectivity, shared weights, and pooling.
Convolutional neural networks: Pooling aggregates local convolutional features into more global representations and improves robustness to small input shifts and distortions.

2.2 Deep generative models and auto-encoders

Deep generative and auto-encoding models build layered representations or generate data through probabilistic, adversarial, and reconstruction-based mechanisms. The survey contrasts their architectures, training procedures, controllability, and stability challenges.

DBNs stack restricted Boltzmann machines and use greedy layer-wise training, while DBMs add bidirectional inference for more ambiguous datasets.
SAEs stack auto-encoders that reconstruct inputs while learning progressively lower-dimensional encoding units.
VAEs replace deterministic hidden representations with parameterizable distributions learned through variational Bayesian inference to generate probabilistic examples.
Generative adversarial networks: GANs use competing generator and discriminator networks to create synthetic images from noise and distinguish generated inputs from real ones.
Generative adversarial networks: Conditional GANs add class labels to generator and discriminator inputs, enabling generation of images from specified classes.
Generative adversarial networks: GANs are frequently affected by training instability and underfitting or overfitting, motivating alternatives such as spectral normalization and WGAN.

2.5 Flow-Based Models

The survey describes flow models as invertible generative systems and situates them alongside autoregressive, convolutional, and recurrent architectures for speech generation and sequential processing.

Flow-Based Models: Flow models use a decoder that is the exact inverse of the encoder, allowing exact sampling from the inferred data distribution.
WaveNet generates audio with deep autoregressive modeling, dilated convolutions, and residual connections, with conditioning enabling text-to-speech generation.
RNNs retain information from past sequence elements, while LSTM gates regulate which information is remembered or forgotten across long separations.

2.8 Attention in Neural Networks

Attention models selectively process relevant portions of visual or sequential inputs instead of entire scenes. The survey reviews attention mechanisms, reinforcement-learning control, and external-memory architectures while noting task-specificity and training instability.

Attention focuses processing on selected visual regions, reducing computational complexity and filtering irrelevant scene information.
Early attention models used saliency maps and recurrent fixation mechanisms to accumulate information about an image over multiple glimpses.
Because the appropriate fixation sequence depends on the input and task, recent attention models often use reinforcement learning to regulate attention.
Reinforcement-learning attention models face policy-design burdens because suitable policies are highly task specific and not generalizable.
Neural Turing machine approaches implement attention with a neural controller and external memory matrix for selective memory access.

2.9 Neural Architecture Search

Neural architecture search automates neural-network design by selecting architectural parameters and connections. Approaches include reinforcement learning, differentiable optimization, and progressive searches that reduce search cost.

Neural architecture search automatically selects neural-network architectural parameters, including convolutional filter sizes, strides, filter counts, and skip connections.
Reinforcement-learning NAS trains an RNN to generate CNN architectural parameters.
DARTS formulates architecture selection as a differentiable function optimized by gradient descent, improving performance over reinforcement learning and reducing search time.
Progressive neural architecture search begins with a simple CNN and searches toward more complex models while reducing search time and space.
The figure search identifies articles showing increasing prominence of deep learning techniques.

3.1 Deep learning in computer vision

Deep learning in computer vision spans CNN-based classification, recognition, and pose estimation, with performance advances from AlexNet through later architectures. The section also emphasizes persistent limitations on context-dependent, arbitrary-view, and unconstrained human-centric tasks.

Image classification and scene labeling: CNNs progressed from handwritten-digit recognition to complex image classification, with AlexNet substantially outperforming earlier state-of-the-art methods on ImageNet.
Image classification and scene labeling: GoogleNet, VGGNet, ZFNet, and ResNet further improved image-recognition performance after AlexNet.
Image classification and scene labeling: ResNet addresses vanishing gradients in deeper CNNs by incorporating residual blocks that combine convolutional operations with skip connections.
Human face, action, and pose recognition: LMCL and ArcFace improve face-recognition objectives by increasing interclass separation while reducing or controlling intraclass variation across benchmarks.
Human face, action, and pose recognition: CNN-based systems support action and pose recognition through multimodal depth-and-skeleton fusion, cascaded refinement, spatial models, adversarial learning, and 3D convolutions.
Limitations: Vision CNNs remain challenged by context-dependent classification, similar actions, arbitrary viewpoints, and unconstrained human poses.

3.2 Deep learning in speech recognition

Deep learning models, especially DBNs and recurrent architectures, achieved strong speech-recognition performance by learning representations and sequential patterns. The section also covers language, emotion, and visual-speech applications, including attention-based approaches.

DBNs achieved a significant milestone in acoustic modeling and were trained layer-wise before end-to-end fine-tuning.
RNNs improve speech recognition by learning sequential patterns, while LSTMs use gates to retain information across longer sequence spans.
A double-layer deep LSTM outperformed a baseline DBN model on a large-vocabulary speech-recognition task.
Deep learning speech applications include sentiment analysis, machine translation, speech emotion recognition, and visual speech recognition.

3.3 Datasets for vision and speech applications

Vision and speech research uses increasingly varied datasets for state-of-the-art benchmarking, spanning large-scale annotated images, unconstrained video-derived imagery, and speech recordings.

ImageNet contains over 14 million labeled images across more than 20,000 categories, while CIFAR-10 contains 60,000 low-resolution images in each of 10 classes.
COCO provides 2.5 million object instances in 328K images for benchmarking tasks including object segmentation and saliency detection.
UAV datasets address detection and tracking under varied weather, obstacles, occlusions, and camera orientations.
Speech benchmarks include TIMIT, with 630 speakers across American English dialects, and VoxCeleb, with transcriptions from 1000 celebrities in unconstrained settings.

3.4 Deep learning in commercial vision and speech applications

Deep learning has become a core research area for major technology companies, while commercial deployment remains constrained by the hardware demands of sophisticated models.

Google, Facebook, Apple, Microsoft, IBM, and other large companies have adopted deep learning as a core AI research area.
Google Brain develops CNN-based architectures using large clusters and high-end GPUs for challenging vision applications.
Personalized and embedded applications remain challenging because robust deep learning algorithms require extensive high-powered, dedicated hardware.

4 VISION AND SPEECH ON RESOURCE RESTRICTED HARDWARE PLATFORMS

Resource constraints make mobile deployment of deep vision and speech models difficult, motivating cloud offloading, compression, and adaptive resource-management approaches. Reported methods reduce memory, computation, or energy demands while retaining useful recognition performance.

State-of-the-art deep models are difficult to adapt to embedded hardware because of large memory footprints and high computational demands.
Mobile devices often offload speech requests to remote servers, introducing latency tied to wireless network quality.
A 1.48-million-parameter DNN outperformed a GMM model while using only 17% of its memory, and LOUDS compression reduced the footprint further by 60%.
Another compressed DNN reduced memory requirements by 75% to 80% to 3.2 MB and computational cost by 10% to 50%, with comparable performance.
Mobile image recognition remains challenging despite progress in mobile speech recognition because of embedded-platform resource constraints.
DeepX decomposes large architectures into smaller blocks and assigns them dynamically to CPUs, GPUs, or LPUs using resource prediction.
Deep compression combines pruning, trained quantization with weight sharing, and Huffman coding to reduce DNN memory footprint.

5 EMERGING APPLICATIONS OF INTELLIGENT VISION AND SPEECH SYSTEMS

Deep vision and speech systems are extending into behavioral analysis, intelligent transportation, and medical imaging. These applications aim to automate measurements, improve prediction and assistance, and support clinical practice.

Behavioral analysis: Vision and speech systems are enabling quantitative analysis of human behavior and expressions across behavioral science, security, forensics, and psychology.Traditional behavioral studies often rely on variable human annotations and qualitative screening, while computational methods support automation and precision.
Behavioral analysis: Facial expressions and speech provide computational signals for studying emotional states, depression severity, and neurodevelopmental disorders.Examples include vocal-idiosyncrasy analysis for depression screening and detection of distinctive speech characteristics associated with autism.
Intelligent transportation: Intelligent transportation systems apply deep learning to driver monitoring, lane tracking, traffic surveillance, flow management, and autonomous driving.CNNs can obtain steering commands from raw front-facing images, while other systems support lane-change, pedestrian, parking, congestion, and traffic prediction tasks.
Precision medicine: Medical imaging systems address the limits of manual visual assessment, where missed subtle markers can contribute to misdiagnosis and delayed diagnosis.The survey presents neural-network-based analysis as promising for clinical measurements, patient-outcome prediction, and physician assistance.

6 LIMITATIONS OF DEEP COMPUTATIONAL MODELS

The survey identifies limitations involving data requirements, computational resources, model interpretability, and overconfidence in replacing human experts. These constraints affect deployment, validation, and the appropriate role of intelligent systems.

Sample size: Deep models require huge training datasets, but medical imaging and behavioral analysis may lack sufficient examples and face expensive data acquisition.High parameter counts can promote overfitting, and some studies report over-optimistic results without proper validation or cross-dataset generalization.
Computational burden: Deep models impose substantial computation, memory, and power demands that restrict deployment on mobile, wearable, and disconnected embedded devices.Real-time learning in memory-constrained applications requires model selection, parameterization, and hardware-software co-design.
Interpretability: Complex architectures and heuristic optimization make deep-model outcomes difficult to interpret, with models often operating as black boxes.The survey uses dropout as an example of randomly deactivating neurons without learning which neurons and weights are most important.
Human expertise: Despite outperforming humans in some applications, intelligent systems should not independently handle critical decisions such as clinical diagnosis.The survey frames neural networks as complementary tools because machines do not master inherently human traits such as empathy.

7 SUMMARY OF SURVEY

This survey synthesizes advances in deep neural architectures, applications, and deployment on constrained platforms. It highlights progress across vision, speech, and emerging interdisciplinary uses while identifying data, hardware, and human-oversight limitations.

Survey scope: The survey reviews state-of-the-art deep neural algorithms and architectures for vision and speech applications.It discusses CNN variants, DBNs, DBMs, VANs, GANs, VAEs, SAEs, RNNs, LSTMs, and hybrid CNN-LSTM systems.
Architectures: CNNs dominate state-of-the-art vision systems, while RNNs and variants such as LSTM lead speech recognition and support image captioning.The survey presents these observations as key findings across industry, academia, speech recognition, NLP, and captioning applications.
Constrained platforms: Compression, quantization, lightweight architectures, and efficiency-oriented software enable deep models to reduce resource requirements and achieve real-time operation on mobile hardware.The survey also identifies hardware-software co-design as necessary for real-time learning in memory-constrained systems.
Emerging applications: Vision and speech systems are driving paradigm shifts in behavioral science, transportation, and clinical research through emotion analysis, autonomous services, and precision diagnostic tools.The survey emphasizes applications that assist or automate aspects of human perception and decision-making.
Limitations and outlook: The survey identifies small-dataset limitations, mobile hardware constraints, and over-optimistic replacement of human experts as major limitations of deep models.It positions the work as a technical resource for future innovations in autonomous systems.