PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey of Convolutions in Deep Learning: Applications, Challenges, and Future Trends

Abolfazl Younesi, Mohsen Ansari, MohammadAmin Fazli, Alireza Ejlali, Muhammad Shafique, Jörg Henkel

arXiv:2402.15490v2cs.LGcs.NE

TL;DR

CNN research spans architectures suited to diverse tasks, but computational complexity and fragmented prior surveys motivate a broader comparative synthesis. This paper surveys CNN types, applications, efficiency methods, frameworks, research fields, challenges, and future trends, concluding that no single convolutional solution is definitive and that context-specific combinations remain important.

  • Problem

    CNNs have diverse architectures and applications, while computational complexity and gaps in earlier surveys limit comprehensive understanding of their trade-offs, strengths, weaknesses, and newer design patterns.

  • Method

    The paper conducts a comprehensive survey and comparison of CNN architectures, applications, efficiency techniques, frameworks, research fields, challenges, and future trends, emphasizing innovations from 2012 onward.

  • Results

    The survey finds that convolution types differ in applications and strengths, while no single solution is definitive; effective choices depend on combining innovations for particular contexts.

  • Takeaways & Limitations

    CNN versatility supports applications beyond traditional computer vision, and future progress is associated with integrating convolutional building blocks with self-supervised learning, attention, and generative models.

Abstract

from arXiv · show

In today's digital age, Convolutional Neural Networks (CNNs), a subset of Deep Learning (DL), are widely used for various computer vision tasks such as image classification, object detection, and image segmentation. There are numerous types of CNNs designed to meet specific needs and requirements, including 1D, 2D, and 3D CNNs, as well as dilated, grouped, attention, depthwise convolutions, and NAS, among others. Each type of CNN has its unique structure and characteristics, making it suitable for specific tasks. It's crucial to gain a thorough understanding and perform a comparative analysis of these different CNN types to understand their strengths and weaknesses. Furthermore, studying the performance, limitations, and practical applications of each type of CNN can aid in the development of new and improved architectures in the future. We also dive into the platforms and frameworks that researchers utilize for their research or development from various perspectives. Additionally, we explore the main research fields of CNN like 6D vision, generative models, and meta-learning. This survey paper provides a comprehensive examination and comparison of various CNN architectures, highlighting their architectural differences and emphasizing their respective advantages, disadvantages, applications, challenges, and future trends.

I. INTRODUCTION

This introduction surveys CNN architectures, applications, efficiency challenges, and research directions while addressing gaps in earlier surveys through an updated taxonomy and broader coverage.

  • CNN foundations: CNNs support computer vision and other applications by extracting local and hierarchical patterns from grid-like data through convolutional operations.Their applications include image recognition, medical imaging, object identification, anomaly detection, audio, and sequential data.
  • Efficiency challenges: Resource-constrained deployment is challenged by CNN computational complexity, motivating lightweight architectures and compression techniques such as MobileNet and TensorFlow Lite.These approaches target computational efficiency, model size, and data-transfer reduction for mobile devices.
  • Survey gap: The survey addresses incomplete prior coverage by classifying newer CNN architectures according to intrinsic design patterns rather than release year.Earlier surveys lacked research questions, dataset analysis, newer model taxonomies, and broad treatment of large vision and language models.
  • Survey scope: The survey focuses on architectural innovations from 2012 onward and discusses recent developments and challenges in greater depth than earlier surveys.The paper presents this updated perspective as useful for researchers studying ongoing CNN development.
  • Survey scope: It analyzes multiple CNN types and compares their architectures, performance, efficiency trade-offs, strengths, weaknesses, applications, and future directions.The stated contributions also include identifying practical model choices, architectural trends, and major CNN research fields.

II. FUNDAMENTALS OF CONVOLUTIONS

Convolutions extract local and multiscale features by sliding learnable kernels across structured data. CNNs organize these operations with pooling and other layers to transform inputs into useful representations for prediction.

  • Convolutional Operations: Convolutions slide kernels across grid-structured data, using element-wise multiplication to capture local patterns such as edges, textures, and shapes.They function as feature extractors for images, videos, and time series.
  • Mathematical Formulation: A 2D convolution combines an input matrix and kernel to produce a feature map through sliding, multiplication, and summation.The same formulation extends to 1D sequences with kernels sliding along one dimension.
  • CNN Architecture: CNNs use convolutional layers with multiple learned filters, followed commonly by pooling layers that downsample feature maps and reduce computational complexity.The architecture also includes input, activation, and fully connected layers for systematic feature extraction and prediction.
  • Wavelet Analysis: Wavelets analyze signals across scales, and CNN filters can learn wavelet-like bases that provide multiresolution representations of input data.Convolutional wavelets have shown promising results in denoising, super-resolution, texture synthesis, segmentation, detection, and classification.
  • Overview: Tables and diagrams summarize CNN components, architectural evolution, and the organization of the survey’s convolutional fundamentals.These references connect convolutional operations with their roles in complete CNN architectures.

A. Background of Deep Learning

Deep learning uses multilayer neural networks to learn hierarchical data representations, while CNNs specialize in extracting spatial features. Basic CNN components support feature learning, dimensionality reduction, nonlinear modeling, and training stability.

  • Deep Learning: Deep learning uses multilayer neural networks to acquire hierarchical representations and discern intricate patterns from raw data.Its development was accelerated by computational power, large datasets, and efficient training algorithms such as backpropagation.
  • CNNs: CNNs are widely used deep-learning models whose convolutional operations support image analysis and feature extraction from intricate structures.Their hierarchical architecture can recognize features in a shift-invariant manner across different image positions and scales.
  • Convolutional Layers: Learned convolutional filters detect patterns, edges, and textures and produce feature maps that highlight regions of interest.These maps support tasks such as identifying category-specific visual patterns.
  • Pooling Layers: Pooling layers reduce feature-map dimensions and computational complexity while introducing translational invariance through max- or average-pooling.Downsampling can improve efficiency and reduce sensitivity to small spatial translations.
  • Activations and Normalization: Activation functions such as ReLU introduce nonlinearity, while Batch Normalization stabilizes and accelerates training by normalizing mini-batch activations.Batch Normalization also permits more aggressive learning rates and can act as a regularizer.

IV. TYPES OF CONVOLUTION IN DEEP LEARNING

CNN convolution types are matched to the structure of the data and the requirements of the application. The survey compares their dimensionality, receptive fields, computational costs, and practical uses.

  • Selection Criteria: Selecting a convolution type depends on the task, dataset, data modality, and computational-efficiency requirements.Different data types may require distinct convolutional designs to capture relevant features, especially in real-time or resource-limited settings.
  • 2D Convolutions: 2D convolutions process image-like grids with sliding kernels and support image recognition, object detection, segmentation, and facial recognition.They capture spatial relationships and hierarchical structures in two-dimensional data.
  • 1D Convolutions: 1D convolutions operate along sequences such as time series, audio, and natural language to detect patterns developing over time.Applications include text classification, sentiment analysis, speaker identification, and emotion recognition.
  • 3D Convolutions: 3D convolutions jointly process spatial and temporal dimensions in volumetric data such as medical scans and videos.They support tumor localization, segmentation, disease classification, and analysis of relationships across scan slices.
  • Point Clouds: Voxelization converts unordered point clouds into volumetric grids so 3D convolutional kernels can process their spatial structure.Sparse, dilated, and multiscale voxel methods adjust receptive fields and capture local and global point features.

D. Dilated Convolutions and Their Advantages

Advanced convolutions modify receptive fields, channel connectivity, sampling, or filtering to improve efficiency and task suitability. The survey relates these designs to applications, architectural evolution, and practical trade-offs.

  • D. Dilated Convolutions and Their Advantages: Dilated convolutions insert gaps between kernel elements to enlarge the receptive field without increasing parameter count.This supports broader contextual capture in semantic segmentation while preserving computational efficiency.
  • Grouped Convolutions: Grouped convolutions divide input and output channels into groups, perform separate within-group convolutions, and concatenate the results.This reduces computational cost and memory consumption while promoting model parallelism and scalability.
  • Architectural Evolution: CNN architectures evolved through multiscale Inception blocks, residual shortcuts, dense feature reuse, depthwise separable convolutions, and systematic EfficientNet scaling.These developments targeted accuracy, trainability, parameter reduction, latency, and efficiency-accuracy trade-offs.
  • Upsampling: Transposed convolutions upsample feature maps and reconstruct higher-resolution representations from lower-resolution inputs for segmentation, generation, and image translation.Overlapping receptive fields can produce checkerboard artifacts, requiring control of stride, padding, and dilation.
  • Depthwise Separable Convolutions: Depthwise separable convolution separates spatial filtering from cross-channel filtering through depthwise and pointwise stages.This design is suited to resource-constrained environments and is used by architectures such as MobileNet and Xception to reduce model size and improve inference speed.
  • Comparative Overview: The comparison tables organize convolution methods by parameters, computational cost, parameter efficiency, upsampling, spatial handling, dependencies, model size, and application characteristics.These axes help relate architectural choices to practical deployment requirements.

C. Spatial Pyramid Pooling (SPP)

Spatial pyramid pooling handles CNN inputs with varying sizes and aspect ratios by pooling across multiple regions and producing fixed-length representations. Attention mechanisms complement convolution by weighting important spatial locations and capturing long-range context.

  • C. Spatial Pyramid Pooling (SPP): SPP handles inputs with varying sizes and aspect ratios by dividing feature maps into regions and pooling each region independently.
  • C. Spatial Pyramid Pooling (SPP): The pooled regional features are concatenated into a fixed-length representation for fully connected layers.This allows CNNs to accept differently sized input images while producing consistent feature maps.
  • C. Spatial Pyramid Pooling (SPP): SPP is useful in object detection and image segmentation applications.
  • C. Spatial Pyramid Pooling (SPP): Attention mechanisms assign weights to spatial locations so convolutions emphasize relevant input regions during feature extraction.Self-attention adaptations can capture long-range dependencies and contextual relationships.

E. Shift-Invariant and Steerable Convolutions

The surveyed convolutional techniques extend CNN capabilities across representation, architecture design, generation, and applications. The section highlights transformation robustness, automated architecture search, adversarial generation, and diverse computer-vision uses.

  • Shift-Invariant and Steerable Convolutions: Shift-invariant convolutions maintain consistent learned features despite small translations, supporting object detection when object locations vary.Steerable convolutions instead use rotatable filters to learn orientation-sensitive features.
  • Capsule Networks: Capsule Networks use neuron groups to represent entity properties such as pose, deformation, and parts, addressing spatial hierarchies and viewpoint variations.Dynamic routing helps model hierarchical relationships and recognize objects across poses, appearances, transformations, and occlusions.
  • Neural Architecture Search: Neural Architecture Search automatically explores convolutional kernel sizes, depths, and connectivity patterns to identify architectures that perform well on specific applications.The search evaluates candidate architectures on validation data through evolution or optimization; autonomous navigation is presented as an example.
  • Generative Adversarial Networks: GANs combine a CNN generator that creates samples with a CNN discriminator that distinguishes real from generated samples through adversarial training.Convolutional GANs support image synthesis, style transfer, data augmentation, and extensions to NLP, audio, video, and medical imaging.
  • Conclusion: Advanced convolutional methods have expanded CNN capabilities in computer vision, image synthesis, and NLP while improving their efficiency, robustness, and expressiveness.The conclusion identifies transposed convolution, capsule networks, NAS, and GANs as examples of this expansion.
  • Applications: Across applications, convolutional techniques support image classification, object detection, image synthesis, and broader computer-vision tasks.Traditional 2D convolutions extract edges, textures, and shapes for classification, while 2D and 3D convolutions process image frames and video for object detection.

C. Natural Language Processing

This section situates CNNs across sequential, volumetric, multimodal, and resource-constrained applications while emphasizing unresolved concerns about efficiency, interpretability, robustness, and domain adaptation.

  • Sequential applications: 1D convolutions extract patterns from sequential text and audio data for tasks including sentiment analysis, named entity recognition, speech recognition, and transcription.
  • Medical imaging: 3D and dilated convolutions are used in medical imaging to process volumetric data and support tumor segmentation, organ localization, and disease classification.
  • Open challenges: Future work addresses CNN interpretability, adversarial robustness, domain knowledge integration, and multimodal scalability.
  • Efficient deployment: Efficient CNN research targets edge and mobile deployment through pruning, compact operators, knowledge distillation, adaptive quantization, and compression.
  • Compression: Filter pruning can remove up to 90% of VGG16 filters without accuracy degradation, while filter-level methods reduce computation by approximately 5x.

E. Multi-Task Learning and Transfer Learning

The section connects CNN research with shared representations, transfer across tasks, optimizer comparisons, and computational evaluation. It frames multi-task learning and transfer learning as responses to limited labeled data and deployment constraints.

  • Multi-task learning: CNNs support multi-task learning by training one model to perform several related applications concurrently.
  • Transfer learning: Shared representations and transferred knowledge can reduce the need for separate large labeled datasets and improve generalization across applications.
  • Limited data: Small-data generalization remains a persistent challenge addressed by transfer learning, few-shot learning, and meta-learning.
  • Multimodal learning: A dual-stream CNN–transformer design uses CNN-extracted visual features to guide language generation and comprehension.
  • Optimizer comparisons: Adam performed well on the reported accuracy and loss comparisons across LeNet-5, VGG16, and ResNet-50 experiments.
  • Resource evaluation: Adam used less CPU than five other optimizers across the three architectures, while its memory usage was among the lowest.

B. Trade-offs between Accuracy and Speed

CNN design requires balancing accuracy, inference speed, memory, computational complexity, and deployment resources. The survey compares optimization techniques, benchmarks, and framework characteristics to support application-specific choices.

  • Accuracy and speed: Higher convolutional complexity can increase inference time while simpler techniques may reduce accuracy, making accuracy–speed balance application-dependent.
  • Experimental comparison: The optimizer comparison uses LeNet-5, VGG16, and ResNet-50 on CIFAR-10 with seven optimizer types and CPU- and memory-utilization measures.
  • Efficiency techniques: Pruning, quantization, and low-rank approximations reduce model size and inference cost, while attention-based convolutions focus computation on important input regions.
  • Memory requirements: Memory demands arise from large parameter sets, intermediate feature maps, and gradients, especially during training and inference on constrained devices.
  • Benchmarking: Standard-dataset benchmarking compares accuracy, inference speed, memory usage, and energy efficiency across convolutional techniques.
  • Practical selection: Benchmarking helps identify strengths and weaknesses and supports selecting techniques for desired performance–efficiency trade-offs.

CPU, GPU

This section surveys deep-learning frameworks and libraries by their programming models, hardware support, usability, speed, and computer-vision capabilities. The comparison highlights different trade-offs between flexibility, performance, deployment, and accessibility.

  • Framework comparison: Table X compares frameworks by release year, supported languages, licenses, model-definition approaches, ease of use, speed, and focus.
  • Frameworks: Caffe introduced a fast, modular layered interface for CNN training with C++ and Python support and a focus on computer vision.
  • Frameworks: TensorFlow provides an end-to-end platform with broad computer-vision support, pretrained models, preprocessing utilities, and object-detection APIs.
  • Frameworks: Keras emphasizes user-friendliness, modularity, and extensibility while providing augmentation tools and predefined computer-vision models.
  • Frameworks: PyTorch emphasizes dynamic neural networks and intuitive Python-based prototyping while supporting GPU/TPU training and production deployment.
  • Libraries: OpenCV supplies traditional computer-vision and image-processing functions and commonly serves as a preprocessing component alongside deep-learning frameworks.
  • Scalable and production tools: MXNet, Chainer, and Deeplearning4j provide combinations of distributed training, GPU support, computer-vision tools, and production-oriented deployment features.

X. MAIN RESEARCH FIELDS

CNNs support diverse computer-vision research fields, from recognition and detection to segmentation, weakly supervised, self-supervised, continual, multimodal, and medical-imaging applications. These fields address limited labels, changing data, multimodal context, and domain-specific structure.

  • Core computer vision tasks: CNNs underpin image classification, object detection, and semantic segmentation through architectures tailored to recognition, localization, and dense pixel labeling.ResNet, Inception, Xception, EfficientNets, Faster R-CNN, YOLO, FCN, U-Net, PSPNet, and DeepLab exemplify these application-specific designs.
  • Learning with limited labels: Few-shot and zero-shot methods seek recognition with one, a few, or no labeled examples by using metric spaces, prototypes, attributes, semantic relationships, or knowledge graphs.These approaches support generalization to new concepts and long-tailed or incremental learning settings.
  • Learning with limited labels: Weakly supervised and self-supervised learning reduce annotation requirements by learning from image-level signals, inferred regions, or unlabeled visual data.Contrastive methods such as SimCLR, SwAV, and MoCo are reported to rival or exceed supervised pre-training in various vision benchmarks.
  • Continual learning: Continual learning addresses non-stationary data by preserving prior knowledge while models learn new classes or changing class definitions.Elastic weight consolidation and incremental moment matching are cited as regularization strategies for reducing catastrophic forgetting.
  • Multimodal and domain applications: Vision-language models combine visual and linguistic inputs for capabilities including zero-shot classification, question answering, visual dialog, captioning, and entity localization.CLIP, ALIGN, and Oscar are identified as large pre-trained examples.
  • Multimodal and domain applications: Medical imaging uses 2D and 3D CNNs for organ segmentation, anomaly localization, and longitudinal patient tracking while incorporating anatomical priors and self-supervised pre-training.The passage emphasizes collaboration between deep-learning methods and domain experts.

K. Video understanding

The surveyed research spans video understanding, multi-task learning, 6D pose estimation, neural architecture search, attention-based alternatives, and generative models. Across these areas, models combine specialized representations, auxiliary objectives, geometric cues, automated design, and synthesis.

  • K. Video understanding: C3D and I3D model video by learning spatial-temporal relationships directly from frame volumes, while captioning and action recognition combine visual, linguistic, and attention mechanisms.Self-supervised learning from unlabeled video repositories is presented as a promising pre-training paradigm.
  • L. Multi-task learning: Multi-task CNNs outperform independent models in low-data regimes by sharing representations across related objectives such as detection, segmentation, counting, keypoints, and surface prediction.The surveyed examples include dense captioning and joint prediction of keypoints, normals, and semantic parts.
  • M. 6D vision: 6D vision estimates an object’s 3D position and orientation from monocular RGB images, with CNN regression, iterative refinement, geometric constraints, and RGB-D or multi-view cues addressing the problem.The monocular setting is challenging because projecting 3D scenes into 2D removes depth information.
  • N. Neural Architecture Search: NAS automates neural-network design by evolving architectures for target datasets and tasks, using search-space design, weight sharing, pruning, predictors, and one-shot methods to reduce search cost.ProxylessNAS searches for efficient mobile architectures directly on target devices.
  • N. Neural Architecture Search: After search, training selected architectures from scratch can improve proxy accuracies, while later evolutionary refinement continues advancing vision models across diverse data, constraints, and objectives.The surveyed NAS approaches also discover non-CNN models for problems beyond computer vision.
  • Attention-based architectures: Attention-based and hybrid vision models replace or combine convolutional filtering with axial or windowed self-attention to capture long-range dependencies while retaining locality or translation-related inductive biases.Examples include NAT, MViT, CoAtNet, and Swin Transformers, with convolution-attention combinations noted for medical segmentation.
  • Generative models: Diffusion and other generative models synthesize detailed images and can provide synthetic training data for downstream classification, detection, and segmentation.DALL-E 2 and DALL-E 3 are cited as large-scale vision diffusion models capable of text-conditioned image generation.

Q. Meta Learning

Meta-learning trains models to adapt rapidly to new tasks from few examples by learning transferable inductive biases, parameter initializations, or task-independent metrics. The broader survey situates this approach among efficient convolutions, attention, generative models, and unresolved robustness and interpretability challenges.

  • Q. Meta Learning: Meta-learning, or learning to learn, develops models that rapidly adapt to new tasks using few examples after training across related tasks.Meta-training learns inductive biases that are applied during meta-testing on novel tasks.
  • Q. Meta Learning: MAML learns an efficient parameter initialization so a few gradient steps can fine-tune the model to new tasks rather than solving one specific task.This provides a model-agnostic approach to fast adaptation in computer vision.
  • Q. Meta Learning: Metric-based approaches represent classes with prototypes and compare new examples in learned similarity spaces, while Meta-Dataset advances evaluation in few-shot image classification.Matching networks are described as using prototype comparisons for fast adaptation.
  • Q. Meta Learning: Self-supervised auxiliary tasks, temporal ensembles, and reinforcement meta-learning further support generalization, robustness to noise, and visuomotor learning from few demonstrations.The cited auxiliary tasks include prediction, rotation, and context modeling.
  • Federated learning: Federated CNN training distributes local updates without exchanging private data, but non-IID distributions, device heterogeneity, and unreliable connectivity remain key challenges.Personalized, multi-task, and meta-learning methods are presented as approaches to statistical heterogeneity.
  • Conclusion: The survey concludes that no single convolutional solution is definitive; effective progress depends on combining innovations for particular applications while refining CNNs with attention, self-supervision, and generative models.Open challenges include robustness, sparse data, interpretability, and trustworthiness.
Loading 2402.15490v2…