Source-linked AI summary
Using Filter Banks in Convolutional Neural Networks for Texture Classification
Vincent Andrearczyk, Paul F. Whelan
TL;DR
Classic CNNs learn useful filter-bank-like features but also emphasize global shape, which is less important for texture analysis. The paper introduces T-CNN, pooling an energy measure from the last convolution layer for texture-focused classification, and reports improved recognition with lower complexity, memory use, and computation.
Problem
Classic CNNs are effective for object analysis but their fully connected layers emphasize overall shape, while the suitability of their learned features for texture analysis lacks direct verification without a sufficiently large texture dataset.
Method
T-CNN pools dense orderless energy descriptors from convolution layers and learns their texture representation and classification through forward and backward propagation.
Results
The T-CNN improves texture-recognition performance while reducing network complexity, memory requirements, and computation, and its combination with a classic CNN improves accuracy further.
Takeaways & Limitations
Compact texture-specific CNNs can achieve similar or better texture-recognition results, while texture and shape processing can be combined within one network.
Takeaways & Limitations
The paper's depth analysis does not generalize to all network architectures, and very deep architectures are outside its scope.
Abstract
from arXiv · showhide
Deep learning has established many new state of the art solutions in the last decade in areas such as object, scene and speech recognition. In particular Convolutional Neural Network (CNN) is a category of deep learning which obtains excellent results in object detection and recognition tasks. Its architecture is indeed well suited to object analysis by learning and classifying complex (deep) features that represent parts of an object or the object itself. However, some of its features are very similar to texture analysis methods. CNN layers can be thought of as filter banks of complexity increasing with the depth. Filter banks are powerful tools to extract texture features and have been widely used in texture analysis. In this paper we develop a simple network architecture named Texture CNN (T-CNN) which explores this observation. It is built on the idea that the overall shape information extracted by the fully connected layers of a classic CNN is of minor importance in texture analysis. Therefore, we pool an energy measure from the last convolution layer which we connect to a fully connected layer. We show that our approach can improve the performance of a network while greatly reducing the memory usage and computation.
1 Introduction
The paper motivates a texture-specific CNN by treating CNN layers as learnable filter banks and reducing the role of global shape information. It targets efficient texture recognition through a compact architecture trained on texture data.
- Texture images exhibit statistical properties and repeated structures that texture-analysis methods exploit for classification, segmentation, and synthesis.
- CNN feature complexity increases with depth, while fully connected layers use convolutional responses to represent overall image shape and class probabilities.
- CNN convolution and pooling layers resemble filter banks, but learn feature responses through forward and backward propagation rather than using only pre-designed filters.
- The proposed network learns texture descriptors and classification jointly, unlike FV-CNN, and evaluates training from scratch alongside texture- and object-pretrained networks.
- Simple networks with fewer neurons and weights achieve similar or better texture-recognition results while targeting lower computational and memory requirements.
2 Related work
Prior work applied standard CNNs to texture recognition, while FV-CNN densely pooled CNN descriptors for texture classification without fine-tuning its CNN on texture data.
- Earlier work used a simple four-layer CNN to classify the Brodatz texture database.
- FV-CNN densely extracts texture descriptors from CNN outputs, replaces fully connected layers with Fisher Vector encoding and an SVM classifier, and supports region recognition.
- The CNN component of FV-CNN uses a pre-trained network whose convolution outputs are not fine-tuned on the texture dataset.
3 Method description
T-CNN treats convolutional layers as learnable filter banks and pools dense, orderless texture descriptors through an energy layer. It can also be integrated with a classic CNN to combine texture and shape information.
- 3.1 Texture Convolutional Neural Network (T-CNN): T-CNN is introduced as a simple architecture for extracting, learning, and classifying dense orderless texture features.
- 3.1 Texture Convolutional Neural Network (T-CNN): The energy layer pools average responses from feature maps, reducing dependence on global spatial information considered less important for texture analysis.
- 3.1 Texture Convolutional Neural Network (T-CNN): T-CNN reduces parameters by removing convolution layers in some variants and using fewer connections between the energy layer E2 and fully connected layer FC3.
- 3.2 Combining texture and classic CNNs: The energy-layer output can be concatenated with the flattened last-convolution output and connected to unchanged fully connected layers.
- 3.2 Combining texture and classic CNNs: TS-CNN-3 integrates T-CNN-3 with a classic CNN to combine texture and overall shape information.
4 Results and analysis
The supplied passage specifies the implementation and training settings used for comparison, including an AlexNet-derived architecture and separate learning rates for training from scratch and fine-tuning.
- The network is implemented in Caffe and derived from AlexNet, retaining comparable feature-map counts and kernel sizes.
- T-CNN fully connected layers can be reduced by more than a factor of two on average without loss of accuracy.
- Training uses base learning rates of 0.001 from scratch and 0.0001 for fine-tuning, with weight decay of 0.0005.
4.2 Datasets
The study evaluates T-CNN across ten datasets, including texture, object, and ImageNet-derived subsets, using dataset-specific resizing, splits, trials, and cross-validation procedures.
- Dataset scope: Ten datasets are used: seven texture datasets and three object datasets.The datasets include DTD, forest species databases, ImageNet and its subsets, kth-tips-2b, Kylberg, and CUReT.
- ImageNet datasets: ImageNet contains 1000 classes, with 1,281,167 training images and 50,000 validation images.The validation set is used for testing, with 50 images per class.
- ImageNet datasets: Three ImageNet subsets contain 28 classes each, comprising texture-like, object-like, or randomly selected classes.Each subset uses the same training and testing images as ImageNet, totaling 1400 test images and approximately 36,000 images per dataset.
- Texture datasets: kth-tips-2b, Kylberg, and CUReT use dataset-specific resampling and evaluation protocols, including four splits, tenfold cross-validation, and 20 repeated splits.Images are resized to 227x227 for kth-tips-2b and CUReT, while Kylberg images are resized to 256x256.
- Texture datasets: DTD provides 47 classes with 120 images each and averages results over ten annotated train-validation-test splits.Images are resized to 227x227 for comparison with AlexNet.
- Texture datasets: Macroscopic and Microscopic forest species databases test larger texture images resized to 640x640, with results averaged over ten trials.Their original image sizes are 3264x2448 and 1024x768, respectively.
4.3 Results of Texture Convolutional Neural Network (T-CNN)
T-CNN achieves similar or better texture-recognition performance than AlexNet with substantially fewer trainable parameters, with three convolution layers generally performing best. Results also show that energy-layer choice, pretraining data, input-size flexibility, and combining texture and shape representations affect performance.
- Networks from scratch and pre-trained: T-CNN-3 performs almost always better than AlexNet across the listed texture datasets while using nearly three times fewer trainable parameters.The comparison covers Kylberg, CUReT, DTD, kth-tips-2b, and ImageNet-T, using networks trained from scratch or pretrained on ImageNet.
- Networks from scratch and pre-trained: 1.7%: T-CNN-3 outperforms AlexNet on kth-tips-2b, achieving 73.2% versus 71.5%.
- Network depth analysis: Three convolution layers provide the best overall T-CNN results, with T-CNN-4 and T-CNN-2 close behind.Using five layers performs worse because the fifth layer extracts sparse, complex object-like features; first-layer features are too simple and lack sufficient learnable parameters.
- Network depth analysis: Shallow T-CNNs perform better with average energy, whereas deeper T-CNNs are more accurate with maximum energy.The maximum energy layer measures whether a feature appears anywhere in the input, without encoding its location or occurrence count.
- Domain transfer: 73.2%: ImageNet pretraining produces the strongest kth-tips-2b result, exceeding T-CNN-3 pretrained on ImageNet-T, ImageNet-S1, or ImageNet-S2.Texture-dataset pretraining reaches 61.8%, while ImageNet-S1 and ImageNet-S2 pretraining reach 56.3% and 59.0%, respectively.
- Results on larger images: +1.4%: T-CNN-3 outperforms the state of the art on the Macroscopic forest species dataset and obtains -0.3% on the Microscopic dataset.The method accepts variable-sized inputs through the energy layer and uses a less complex approach than the comparison method.
4.4 Results combining texture and shape analysis
Combining T-CNN texture representations with AlexNet shape representations improves classification on kth-tips-2b. The integrated TS-CNN achieves the strongest reported result while using fewer parameters than the compared FV/FC-CNN approach.
- Combined texture and shape network: 73.7% accuracy is achieved by the combined TS-CNN, the best result reported for the tested texture-and-shape integration.TS-CNN combines the T-CNN texture approach with a classic deep neural network to learn texture and overall shape information.
- Combined texture and shape network: Adding AlexNet voting to T-CNN-3 increases kth-tips-2b classification by 0.7% over T-CNN-3 and 1.6% over AlexNet.The voting scores are summed to provide an averaged classification from the two networks.
- Model efficiency: The texture, shape, and combined approaches obtain similar results, while the proposed algorithm remains fully contained in a CNN architecture with fewer parameters.The comparison is described for the results reported in Table 6.
5 Conclusion
The paper introduces T-CNN, a texture-focused CNN architecture inspired by neural networks and filter banks. It reports improved texture-recognition performance with lower complexity, memory requirements, and computation, and shows complementarity between texture and shape analysis.
- Conclusion: T-CNN introduces an energy measure that discards overall shape information analyzed by classic CNNs.The architecture is inspired by classic neural networks and filter-bank approaches.
- Conclusion: T-CNN increases texture-recognition performance while reducing complexity, memory requirements, and computation time.The conclusion states these reductions are large, without giving specific values here.
- Conclusion: A network combining T-CNN with a classic deep neural network demonstrates complementarity between texture and shape analysis with a significant accuracy improvement.The combined network incorporates both approaches within one architecture.
6 Future research
The energy-pooling approach supports inputs with different image dimensions without changing the network structure. Future work could use multiple rescaled inputs or incorporate GoogLeNet's multiscale analysis and computation reduction.
- Future research: Energy pooling allows images of various dimensions to use the exact same network structure.This makes the approach suitable for multiscale processing with differently sized inputs.
- Future research: Multiple rescaled images could be used as inputs to implement multiscale analysis.The proposed approach keeps the same structure while processing these rescaled inputs.
- Future research: GoogLeNet's multiscale analysis and computation reduction could alternatively be incorporated into the network.The passage presents this as an alternative future direction.
Appendix A: List of classes from ImageNet in ImageNet-T
Appendix A lists ImageNet classes included in ImageNet-S1, including categories such as shops, objects, landscapes, and foods. The supplied passages provide class names and identifiers rather than analysis of their classification performance.
- ImageNet-S1 classes: The appendix lists ImageNet class identifiers and names, including bookshops, butcher shops, chains, chainlink fences, and cliff dwellings.These entries appear in the supplied ImageNet-S1 class list.
- ImageNet-S1 classes: Additional listed classes include Chihuahua, golden retriever, ladybug, acoustic guitar, ambulance, clocks, and household objects.The entries are presented as ImageNet identifiers paired with class names.
- ImageNet-S1 classes: The list also includes folding chairs, fountain pens, hammers, iPods, lens caps, motor scooters, park benches, plungers, and sunglasses.These categories continue the appendix's ImageNet class enumeration.
Appendix C: List of classes from ImageNet in ImageNet-S2
The appendix lists ImageNet classes included in ImageNet-S2, spanning animals, objects, and other everyday items.
- Listed animal classes include common iguana, partridge, flat-coated retriever, bull mastiff, miniature poodle, cheetah, sea urchin, orangutan, and gibbon.
- Listed object classes include Band Aid, bathing cap, chain saw, container ship, hook, hourglass, and jinri.