Source-linked AI summary
Binary Patterns Encoded Convolutional Neural Networks for Texture Recognition and Remote Sensing Scene Classification
Rao Muhammad Anwer, Fahad Shahbaz Khan, Joost van de Weijer, Matthieu Molinier, Jorma Laaksonen
TL;DR
Robust texture recognition and remote sensing scene classification require features that handle realistic imaging variation, while common CNNs primarily use RGB inputs. The paper introduces LBP-based TEX-Nets with early and late RGB–texture fusion. Late fusion consistently improves standard RGB performance on both tasks, and the final combination outperforms the state of the art for remote sensing scene classification.
Problem
Designing texture features robust to realistic imaging conditions is challenging, while CNN practice primarily uses RGB inputs for texture and remote sensing recognition.
Method
The paper constructs TEX-Nets by adding LBP-based texture-coded mapped images as a second stream and evaluates early and late fusion with RGB.
Results
Late fusion always improves the standard RGB architecture on both recognition tasks, and the final combination improves results over the state of the art for remote sensing scene classification.
Takeaways & Limitations
LBP-based texture streams provide complementary information to RGB deep models within the evaluated recognition tasks.
Takeaways & Limitations
The evaluated approach is bounded by RGB imagery and does not address satellite images containing additional spectral bands such as Near Infrared.
Abstract
from arXiv · showhide
Designing discriminative powerful texture features robust to realistic imaging conditions is a challenging computer vision problem with many applications, including material recognition and analysis of satellite or aerial imagery. In the past, most texture description approaches were based on dense orderless statistical distribution of local features. However, most recent approaches to texture recognition and remote sensing scene classification are based on Convolutional Neural Networks (CNNs). The d facto practice when learning these CNN models is to use RGB patches as input with training performed on large amounts of labeled data (ImageNet). In this paper, we show that Binary Patterns encoded CNN models, codenamed TEX-Nets, trained using mapped coded images with explicit texture information provide complementary information to the standard RGB deep models. Additionally, two deep architectures, namely early and late fusion, are investigated to combine the texture and color information. To the best of our knowledge, we are the first to investigate Binary Patterns encoded CNNs and different deep network fusion architectures for texture recognition and remote sensing scene classification. We perform comprehensive experiments on four texture recognition datasets and four remote sensing scene classification benchmarks: UC-Merced with 21 scene categories, WHU-RS19 with 19 scene classes, RSSCN7 with 7 categories and the recently introduced large scale aerial image dataset (AID) with 30 aerial scene types. We demonstrate that TEX-Nets provide complementary information to standard RGB deep model of the same network architecture. Our late fusion TEX-Net architecture always improves the overall performance compared to the standard RGB network on both recognition problems. Our final combination outperforms the state-of-the-art without employing fine-tuning or ensemble of RGB network architectures.
1. Introduction
The paper addresses robust texture recognition and remote sensing scene classification by integrating LBP-based texture information into CNN architectures alongside RGB inputs. It proposes TEX-Nets with early and late fusion, reporting that late fusion improves standard RGB performance and that the final combination exceeds the state of the art for remote sensing scene classification.
- Research problems: Robust texture description remains challenging under variations in scale, orientation, illumination, and other imaging conditions.
- Research problems: Remote sensing scene classification must handle large intra-class variation, illumination changes across times and seasons, small inter-class dissimilarity, and scale variation.
- Prior approaches: CNN-based recognition commonly uses RGB image patches and large labeled datasets such as ImageNet, while LBP variants can remain competitive under rotations and several noise types.
- Proposed approach: The paper integrates LBP within deep architectures for texture recognition and remote sensing scene classification, producing models called TEX-Nets.
- Proposed approach: TEX-Nets use texture-coded mapped images as a second stream alongside RGB, with early fusion joining channels at the input and late fusion combining separately trained streams at the fully connected layers.
- Findings: Late fusion consistently outperforms early fusion and improves the standard RGB architecture, while the final combination surpasses the state of the art for remote sensing scene classification without RGB fine-tuning or architectural ensembling.
2. Related Work
Prior work spans handcrafted texture descriptors, deep CNN features, and multimodal fusion, while remote-sensing studies commonly use patch-based, ImageNet-pretrained networks. This paper investigates combining explicit binary-pattern texture information with RGB deep models through two-stream fusion architectures.
- Texture recognition: LBP and its variants remain widely used for texture recognition, with extensions addressing rotation invariance, noise tolerance, variance, phase, and ternary patterns.Related work also combines LBP descriptors with color features.
- Deep learning: CNNs typically process fixed-sized RGB images through convolution, normalization, pooling, and fully connected layers, after training on large labeled datasets.Intermediate fully connected activations have also been used as recognition features.
- Remote sensing image analysis: Deep learning is widely applied to remote-sensing analysis, including optical, hyperspectral, and SAR imagery, but most studies train on patches rather than full-sized satellite images.Full-image training is constrained by the memory demands of images measuring one to several gigabytes.
- Remote sensing image analysis: Earlier remote-sensing work often relied on ImageNet-pretrained networks because large, high-quality labeled satellite-image datasets are difficult to obtain.This creates a setting in which RGB deep features are transferred from general computer-vision benchmarks.
- Texture recognition: Deep texture-recognition methods include scattering networks, PCANet, Fisher Vector encoding of CNN descriptors, and other learned deep features.These approaches differ in whether filters are fixed, learned through multistage analysis, or extracted from pretrained networks.
- Our Approach: The paper proposes texture-coded mapped images as a second stream alongside RGB and investigates early and late fusion for texture recognition and remote-sensing scene classification.It presents the two-stream fusion investigation as novel in these tasks and evaluates the approach on four remote-sensing benchmarks.
3. Binary Patterns Encoded Convolutional Neural Networks
This section introduces TEX-Nets, which transform LBP texture codes into CNN-compatible mapped images and fuse texture with RGB information. Experiments compare early and late fusion across architectures and show late fusion consistently outperforms RGB and early fusion.
- LBP texture coding: LBP describes local texture by thresholding neighborhood pixels around a center pixel into binary codes.With an 8-pixel neighborhood, codes range from 0 to 255; histograms of codes provide translation-normalized features invariant to monotonic grayscale changes.
- Motivation: Directly feeding unordered LBP codes to CNNs is unsuitable because convolution performs weighted averaging over input values.The paper therefore maps LBP codes into a metric space before convolution.
- Two-stream architectures: TEX-Nets combine texture-coded mapped images with raw RGB inputs through two-stream architectures fused at different stages.The study evaluates early fusion and late fusion using VGG-M and ResNet-50 networks.
- Results: 34.4% top-1 error is achieved by late fusion versus 35.3% for six-channel early fusion on ILSVRC 2012.Late fusion also provides superior results compared with both standard RGB and early fusion, while increasing parameters by a factor of 1.4.
4. Experimental Results
Experiments evaluate TEX-Nets on four texture datasets and four remote sensing benchmarks, comparing texture-coded, RGB, and early- or late-fusion architectures. Late fusion consistently outperforms the corresponding RGB baselines and improves results against existing methods across the evaluated tasks.
- Experimental setup: TEX-Net models use texture-coded mapped images, while the experimental setup also evaluates standard RGB networks and early- and late-fusion two-stream architectures.The VGG-M models are trained on ImageNet and evaluated without fine-tuning, using FC7 features and a linear SVM for texture recognition.
- Texture recognition: On DTD, early fusion reaches 64.0% and 64.6% mean classification scores versus 63.4% for the standard RGB VGG-M network.The TEX-Net standard model is also evaluated on the same dataset, alongside the two early-fusion variants.
- Texture recognition: On Texture-10, late fusion with VGG-M reaches 91.3% mean accuracy, a 4.0% gain over the standard RGB VGG-M network at 87.3%.The best Texture-10 results use the late-fusion approach with ResNet.
- Remote sensing scene classification: Across remote sensing datasets, late-fusion ResNet consistently outperforms standard RGB ResNet and improves over existing high-level methods, with large gains on RSSCN7 and AID.The reported comparisons include consistent gains across all four remote sensing datasets and improved performance on most scene categories.
5. Conclusions
The paper concludes that combining texture-coded mapped images with RGB in a late-fusion two-stream architecture improves recognition performance. It identifies broader texture encodings, fusion strategies, and full-sized multispectral satellite imagery as future directions.
- Conclusions: The paper investigates LBP-encoded CNNs and early- and late-fusion architectures for texture recognition and remote sensing scene classification.The models construct a texture stream from coded mapped images and combine it with a standard RGB stream.
- Conclusions: Experiments on benchmark texture and remote sensing datasets show that late fusion always improves overall performance over the standard RGB stream for both recognition tasks.The final combination also improves results compared with the state of the art for remote sensing scene classification.
- Future work: Future work includes alternative texture-description techniques, additional fusion strategies, and training and testing on full-sized satellite images with spectral bands beyond RGB.Near Infrared is given as an example of an additional spectral band.