Source-linked AI summary
Impact of Fully Connected Layers on Performance of Convolutional Neural Networks for Image Classification
S. H. Shabbeer Basha, Shiv Ram Dubey, Viswanath Pulabaigari, Snehasis Mukherjee
TL;DR
The paper studies how CNN architecture choices relate to fully connected layers, addressing limited exploration of architecture selection. Experiments with multiple CNN models examine FC-layer design and report different requirements for shallow and deep CNNs.
Problem
Architecture selection for better CNN performance remains insufficiently explored, while careful selection of network width and FC layers is needed.
Method
The study compares four CNN models, including three plain architectures, while varying FC-layer configurations with added layers using ReLU, batch normalization, and dropout by default.
Results
Deeper CNNs perform better than shallow models over deeper datasets, while shallow CNNs require more FC-layer nodes and deeper CNNs need fewer FC layers.
Takeaways & Limitations
CNN depth and FC-layer design should be considered together when selecting architectures for datasets.
Takeaways & Limitations
The paper defines one CNN as deep or shallow relative to another according to the number of trainable layers.
Abstract
from arXiv · showhide
The Convolutional Neural Networks (CNNs), in domains like computer vision, mostly reduced the need for handcrafted features due to its ability to learn the problem-specific features from the raw input data. However, the selection of dataset-specific CNN architecture, which mostly performed by either experience or expertise is a time-consuming and error-prone process. To automate the process of learning a CNN architecture, this paper attempts at finding the relationship between Fully Connected (FC) layers with some of the characteristics of the datasets. The CNN architectures, and recently datasets also, are categorized as deep, shallow, wide, etc. This paper tries to formalize these terms along with answering the following questions. (i) What is the impact of deeper/shallow architectures on the performance of the CNN w.r.t. FC layers?, (ii) How the deeper/wider datasets influence the performance of CNN w.r.t. FC layers?, and (iii) Which kind of architecture (deeper/ shallower) is better suitable for which kind of (deeper/ wider) datasets. To address these findings, we have performed experiments with three CNN architectures having different depths. The experiments are conducted by varying the number of FC layers. We used four widely used datasets including CIFAR-10, CIFAR-100, Tiny ImageNet, and CRCHistoPhenotypes to justify our findings in the context of the image classification problem. The source code of this research is available at https://github.com/shabbeersh/Impact-of-FC-layers.
1. Introduction and Related Works
The paper studies how fully connected layers interact with CNN depth and dataset characteristics in image classification. It targets limited evidence on selecting FC-layer configurations and architecture depth for different datasets.
- The study analyzes FC-layer effects on CNN image-classification performance alongside CNN depth and dataset depth.
- The paper addresses limited prior analysis of FC-layer hyperparameters, including FC-layer count and neuron count, for CNN architecture design.
- It systematically evaluates deeper and shallower CNN architectures while varying the number of FC layers.
- It examines how deeper and wider datasets affect CNN performance with different FC-layer configurations.
- Experiments cover natural-image datasets CIFAR-10, CIFAR-100, and Tiny ImageNet, plus the biomedical CRCHistoPhenotypes dataset.
2. Developed CNN Architectures
The paper defines CNN depth by trainable-layer count and develops four models to study FC-layer effects, including three plain architectures and one with skip connections. The models adapt AlexNet-, VGG-, and ResNet-inspired designs to the target datasets.
- The study implements four CNN models: three plain architectures and one with ResNet-style skip connections.
- Deep and Shallow CNNs: CNN depth is defined by comparing the number of trainable layers between models.
- CNN-1 Architecture: CNN-1 adapts AlexNet for low-resolution images and uses five convolutional layers before its FC components.
- Output FC layers contain 10, 100, 200, or 4 neurons for CIFAR-10, CIFAR-100, Tiny ImageNet, and CRCHistoPhenotypes, respectively.
- CNN-2 Architecture: CNN-2 is based on CIFAR-VGG, removes some convolutional layers, and contains five two-convolution blocks followed by a pooling layer and an output FC block.
- CNN-3 Architecture: CNN-3 uses the CIFAR-VGG model as a third deep network for studying FC-layer impact.
3. Experimental Setup
The experiments evaluate CNN classification accuracy across natural and biomedical image datasets while varying FC layers under standardized training and regularization settings. Dataset depth and width are defined using images per class and number of classes.
- Experiments use CIFAR-10, CIFAR-100, Tiny ImageNet, and CRCHistoPhenotypes across natural and biomedical image modalities.
- Training details: Added FC layers use ReLU, batch normalization, and dropout by default, with data augmentation also applied to reduce over-fitting.
- Training details: The models are trained for 250 epochs with SGD, momentum 0.9, and a learning rate initially set to 0.1.
- Classification accuracy is used as the performance metric for CNN-1, CNN-2, and CNN-3.
- Deeper vs Wider datasets: For datasets with roughly equal image counts, a deeper dataset has more training images per class, while a wider dataset has more classes and fewer images per class.
- Deeper vs Wider datasets: CIFAR-10 is treated as deeper than CIFAR-100 because both have 50,000 training images, but CIFAR-10 has 5,000 images per class versus 500 for CIFAR-100.
4. Results and Analysis
Experiments across four image datasets varied CNN depth and FC-layer structure to examine how architecture and dataset characteristics affect classification. Deeper architectures generally needed fewer FC layers and neurons, while dataset width influenced which architecture performed better.
- Cross-dataset findings: Across datasets, deeper architectures did not require more FC layers, whereas shallow architectures required more FC layers for better performance.The authors attribute this behavior to differences in the features learned by convolutional layers.
- Dataset width and depth: CNN-1 required more FC neurons for wider datasets, whereas CNN-3 required fewer FC neurons for wider datasets than for deeper datasets.The comparison is presented as an effect of dataset type on FC-layer requirements.
- CIFAR-10 analysis: 92.29% was CNN-2’s best CIFAR-10 accuracy with three FC layers containing 4096, 256, and 10 neurons, versus 92.02% with two layers.The two-layer configuration used 256 and 10 neurons.
5. Conclusion
The paper concludes that CNN depth, dataset type, and FC-layer design should be considered together when selecting an image-classification architecture. Its guidelines link deeper CNNs with deeper datasets and shallow CNNs with wider datasets.
- Conclusion: Careful selection of CNN architecture and FC layers can improve model performance and reduce architecture-selection time.The conclusion frames these decisions as choices among deeper and shallow architectures.
- Conclusion: Deeper CNNs need fewer FC neurons irrespective of dataset type.This is stated as a separate FC-layer design guideline.
- Conclusion: Shallow CNNs require more FC neurons and FC layers for wider datasets than deeper CNNs.This guideline specifically connects shallow models with wider datasets.
- Conclusion: Deeper CNNs perform better than shallow models over deeper datasets, whereas shallow architectures perform better over wider datasets.The conclusion presents this as a practical architecture-selection guideline.