Source-linked AI summary
From BoW to CNN: Two Decades of Texture Representation for Texture Classification
Li Liu, Jie Chen, Paul Fieguth, Guoying Zhao, Rama Chellappa, Matti Pietikainen
TL;DR
Texture representation is central to image understanding but must handle substantial variation, limited data, and competing accuracy-efficiency requirements. This paper surveys two decades of BoW-, CNN-, and attribute-based methods, datasets, and benchmark performance, then discusses open challenges and future directions. Its conclusion frames practical texture representation as a compromise among accuracy, robustness, compactness, available training data, and implementation cost.
Problem
Texture representation remains a fundamental and challenging problem because real textures vary substantially while useful labeled data and computational resources are limited.
Method
The paper provides a comprehensive survey of post-2000 texture representation, organizing methods into BoW-based, CNN-based, and attribute-based categories and reviewing datasets and evaluations.
Results
The survey documents extensive progress in texture representation, including state-of-the-art SRP classification rates of 99.4% on CUReT, 99.3% on KTHTIPS, and 99.3% on UMD.
Takeaways & Limitations
Practical texture representation requires balancing classification accuracy, robustness to image degradations, compactness, efficiency, training-data availability, and implementation cost.
Takeaways & Limitations
CNN descriptors can require very high computational complexity and memory, while robustness to image degradation remains a limitation.
Abstract
from arXiv · showhide
Texture is a fundamental characteristic of many types of images, and texture representation is one of the essential and challenging problems in computer vision and pattern recognition which has attracted extensive research attention. Since 2000, texture representations based on Bag of Words (BoW) and on Convolutional Neural Networks (CNNs) have been extensively studied with impressive performance. Given this period of remarkable evolution, this paper aims to present a comprehensive survey of advances in texture representation over the last two decades. More than 200 major publications are cited in this survey covering different aspects of the research, which includes (i) problem description; (ii) recent advances in the broad categories of BoW-based, CNN-based and attribute-based methods; and (iii) evaluation issues, specifically benchmark datasets and state of the art results. In retrospect of what has been achieved so far, the survey discusses open challenges and directions for future research.
1 Introduction
Texture is a fundamental visual cue across diverse images and a longstanding challenge in computer vision and pattern recognition. This survey addresses the need for an updated, broad review of texture representation since 2000.
- Texture describes the spatial organization of basic elements or textons, ranging from stochastic to perfectly regular patterns.
- Texture analysis supports applications including medical imaging, quality inspection, and content-based image retrieval.
- Texture representation extracts features describing texture information and forms the core of texture analysis.
- Existing surveys largely covered methods before 2000 or focused exclusively on Local Binary Patterns, leaving no extensive survey of post-2000 advances.
- The paper reviews core tasks, challenges, representative taxonomies, texture datasets, and state-of-the-art performance.
2 Background
Texture classification assigns images or regions to known categories, but effective representation must handle substantial appearance variation, subtle class differences, and limited data. The field progressed from statistical and filtering models toward texton, BoW, and invariant local-descriptor approaches.
- 2.1 The Problem: Texture classification determines whether an image or region belongs to one of several known texture categories, including binary medical hypotheses.
- 2.1 The Problem: Powerful texture features are critical because even the best classifier performs poorly when the representation is inadequate.
- 2.2 Summary of Progress in the Past Decades: Historical approaches evolved from cooccurrence statistics, filtering, and statistical modeling to texton dictionaries, BoW histograms, and local invariant descriptors.
- 2.2 Summary of Progress in the Past Decades: Texture representations have also transferred to object recognition, scene classification, and facial image analysis, narrowing the distinction between texture-specific and generic descriptors.
- 2.1 The Problem: Texture representation must balance high-quality, robust features against high efficiency and real-time computational requirements.
- 2.1 The Problem: Texture recognition is difficult because illumination, rotation, scale, blur, noise, occlusion, and viewpoint changes create large intraclass variations and small interclass differences.
3 Bag of Words based Texture Representation
BoW-based texture representation converts local texture information into a global feature representation, while the field also faces scalability and deployment constraints. Texton theory provides a conceptual foundation for these representations.
- Texture representation transforms an input image into a feature vector describing texture properties for subsequent tasks such as classification.
- Because texture is spatial, representations analyze local pixel neighborhoods before aggregating local features into an image- or region-level representation.
- Efficient representations are needed for many texture categories, high-dimensional features, large-scale problems, and resource-limited devices.
- Textons are elementary units of preattentive texture perception, although early studies focused on artificial patterns and lacked a rigorous definition.
3.1 The BoW Pipeline
The BoW pipeline processes an image through local patch extraction and description, codebook generation, feature encoding, pooling, and classification. Each stage affects the quality, efficiency, or final discriminative power of the representation.
- BoW begins by extracting N patches from interest points, a fixed grid, or dense pixel positions.
- Local descriptors map the extracted patches to D-dimensional texture features that should be distinctive and robust to image transformations.
- A codebook of K D-dimensional codewords is learned from training data or predefined, and its size and structure affect discrimination and efficiency.
- Feature encoding maps each local feature to one or more codewords, producing coding vectors that link local representation with feature pooling.
- Pooling aggregates coded local vectors into a global representation, which classifiers such as SVMs, nearest neighbors, neural networks, or random forests use for prediction.
3.2 Local Texture Descriptors
Local texture descriptors are organized as sparse or dense representations, with methods designed to provide invariance to imaging changes. The surveyed approaches combine detectors, descriptors, filters, and learning strategies, with performance depending on the task and variation conditions.
- Descriptor organization: Local descriptors are categorized by whether they are applied densely at every pixel or sparsely at selected interest points.Both approaches aim for invariance to contrast, rotation, scale, and possibly other criteria.
- Sparse Texture Descriptors: Sparse descriptors detect and geometrically normalize stable affine-covariant regions before encoding their local appearance.Examples include Harris affine and Laplacian blob detectors paired with SPIN and RIFT descriptors.
- Sparse Texture Descriptors: 95.3% and 98.7% classification rates were reported on CUReT and UIUC using complementary region detectors, local descriptors, and SVM classification.The study recommended combining complementary features while avoiding more invariance than the application requires.
- Dense Texture Descriptors: 99.4%, 99.3%, and 99.3% classification rates were reported for SRP on CUReT, KTHTIPS, and UMD, respectively.SRP uses sorted local features and random projections to obtain lower-dimensional representations for robust texture classification.
- Dense Texture Descriptors: LBP is effective and efficient with rotation variation and limited illumination change, but its performance degrades under strong illumination, affine transformations, or noise.MRELBP reached 100% on Outex TC10 and 99.8% on Outex TC12, while LBP-based features were especially effective with limited training data.
- Fractal Based Descriptors: Wavelet-based MFS improved classification to 98.6% on UIUC and 98.7% on UMD, although MFS requires high-resolution images for stable features.The original MFS achieved 92.3% on UIUC and 93.9% on UMD and had limited discriminability because of its simple, low-dimensional features.
3.3 Codebook Generation
Codebook generation converts redundant local texture features into prototype-based representations for encoding. Learned codebooks commonly use k-means or GMMs, while reconstruction-based methods learn dictionaries that support sparse representations and reduce reconstruction error.
- Local texture features are redundant, motivating prototype features such as codewords or textons for representation.
- LBP and BIF use predefined codebooks and therefore bypass codebook learning.
- K-means and Gaussian Mixture Models are the most popular learned-codebook methods.GMMs model cluster centers and covariances, whereas k-means considers distances to cluster centers and cannot capture overlapping distributions in the same way.
- Reconstruction-based methods learn nonparametric redundant dictionaries that enable sparse data representations and minimize reconstruction error.
- FV extends BoW by encoding first- and second-order statistics, while BoW captures zero-order statistics and VLAD accumulates feature-to-codeword differences.
- Codebook generation is often interleaved with feature encoding because the codebook provides the basis for encoding feature vectors.
3.4 Feature Encoding
Feature encoding transforms local texture descriptors into vectors that can be aggregated into global image representations. The survey organizes encoding into voting-based, Fisher Vector-based, and reconstruction-based strategies.
- Voting based: Voting-based methods assign local descriptors to codewords and aggregate them into a K-dimensional histogram.Hard voting uses the nearest codeword, while soft voting distributes weights across several nearby codewords according to distance.
- Fisher Vector based: Fisher Vector methods encode higher-order distribution information beyond BoW’s zeroth-order codeword counts.IFV uses a GMM soft codebook and gradients with respect to mixture parameters, producing 2DK-dimensional encodings.
- Fisher Vector based: IFV can use smaller codebooks than BoW and often achieves good performance with linear classifiers, but its vectors are substantially higher-dimensional.Typical IFV codebooks contain 64 or 256 components, while resulting vectors can reach tens of thousands of dimensions.
- Reconstruction based: Reconstruction-based methods seek information-preserving encodings that reconstruct local features using few codewords.Sparse coding uses sparse decompositions, whereas LLC projects descriptors onto the subspace spanned by their nearest q codewords.
- Reconstruction based: LLC produces a K-dimensional encoding whose nonzero entries correspond to the q codewords closest to the local descriptor.A small regularization constant controls the projection, while σ adjusts weight-decay speed.
3.5 Feature Pooling and Classification
Feature pooling combines coded local descriptors into compact global representations, while spatial pooling preserves some location information. The resulting pooled features can be classified using standard classifiers and distance measures.
- Feature Pooling: Pooling integrates coded feature vectors into a compact representation intended to improve robustness to image transformations and noise.Common operations include sum, average, and max pooling; max pooling may suit sparse features particularly well.
- Feature Pooling: Applying sum, average, or max pooling reduces N coded features of dimension d to a single d-dimensional feature vector.The operation is applied row-wise to the coded-feature matrix.
- Feature Pooling: Spatial Pyramid Pooling partitions images into increasingly fine subregions and concatenates pooled regional histograms to preserve spatial information.This produces a higher-dimensional representation than pooling over the entire image.
- Classification: Pooled BoW features are commonly classified with Nearest Neighbor Classifiers or Support Vector Machines.EMD, KL divergence, and Chi Square distance are among the measures used for comparing histogram representations.
4 CNN based Texture Representation
CNN-based texture representations are surveyed across pretrained generic models, finetuned models, and handcrafted deep convolutional networks. CNN features can be combined with traditional encoders, while orderless pooling and dataset scale strongly shape performance and transferability.
- 4 CNN based Texture Representation: CNN texture representations are grouped into pretrained generic CNNs, finetuned CNNs, and handcrafted deep convolutional networks.These categories organize the survey’s treatment of CNN-based texture representation methods.
- CNN architecture: CNNs learn filters directly from data through convolution, nonlinearities, and pooling, forming increasingly abstract hierarchical representations.These operations resemble traditional filter-bank methods but use learned filters and hierarchical processing.
- Using Pretrained Generic CNN Models: CNN transfer performance varies by layer, with fully connected layers generally showing weaker generalization and transferability than other layers.The survey notes that fully connected layers primarily serve classification and may transfer less effectively to texture tasks.
- Using Pretrained Generic CNN Models: IFV was the best-performing encoder among BoW, LLC, VLAD, and IFV when evaluated on CNN features.VGGVD+IFV with a linear SVM achieved near-perfect results on several texture datasets, including 99.8% on KTHTIPS and 99.9% on UIUC and UMD.
- CNN pooling: Bilinear CNN and Fisher-vector CNN descriptors showed virtually identical texture-classification performance in reported comparisons.Bilinear pooling uses outer products of convolutional features, yielding descriptors of dimension (D_l)^2.
- Using Finetuned CNN Models: Finetuned CNNs generate global representations end-to-end, avoiding the explicit encoding and pooling stages used in pretrained CNN pipelines.Finetuning modifies the final fully connected layer to contain B nodes corresponding to target classes.
- Using Finetuned CNN Models: Large-scale texture datasets comparable to ImageNet or MITPlaces are absent, limiting exploration of finetuning large CNNs or training them from scratch.Most existing texture datasets are small, and reported finetuning of VGGNet or AlexNet produced negligible improvement on them.
- Using Handcrafted Deep Convolutional Networks: PCANet extracts features faster than ScatNet but has weaker invariance and texture-classification performance.The handcrafted networks use either random filters or filters learned through Linear Discriminant Analysis.
5 Attribute-Based Texture Representation
Attribute-based texture representation describes textures with semantic properties rather than only category labels. The survey covers attribute vocabularies, annotated datasets, and methods for estimating attributes from low-level representations.
- Motivation: Visual texture attributes provide more detailed descriptions than category labels and are linked to broader scene-understanding goals.Examples include mesh, spotted, striated, and striped descriptions of textured objects.
- Essential Issues: Three core issues are identifying a universal attribute vocabulary, building an attribute-annotated benchmark, and reliably estimating attributes from images.Estimation can use low-level BoW- or CNN-based texture representations.
- Attribute Vocabulary: Texture-attribute vocabularies have evolved from manually proposed properties to psychologically derived adjective clusters.Examples include Tamura’s six attributes, Amadasun and King’s five attributes, and eleven clusters identified from 98 adjectives.
- Attribute Annotation: Pairwise comparison was proposed as an alternative way to collect human judgments about texture attributes.The framework presents two texture images and asks a subject to choose which exhibits more of an attribute.
- Attribute Datasets: Cimpoi et al. selected 47 English texture attributes and collected 120 example images for each attribute.Their comparisons of BoW- and CNN-based methods showed that attributes transferred between datasets.
6 Texture Datasets and Performance
Texture benchmarks evolved from controlled, often saturated datasets toward more realistic collections with substantial variation in material instances, pose, illumination, and scale. CNN descriptors with IFV encoding deliver strong results, while invariant handcrafted descriptors remain competitive under degradation and resource constraints.
- Benchmark datasets: Datasets have become progressively more challenging, serving both as common performance measures and as drivers of increasingly difficult recognition problems.
- Benchmark datasets: Brodatz and VisTex provide early single-image-per-class benchmarks, but their limited intraclass variation constrains evaluation realism.Brodatz is criticized for lacking scale, rotation, perspective, and illumination variation; VisTex is rarely used for image-level classification.
- Benchmark datasets: CUReT introduced real-world appearance variation through specularities, interreflections, shadowing, and surface-normal changes, but has limited scale change and in-plane rotation.
- Benchmark datasets: KTHTIPS and KTHTIPS2b extended texture evaluation with variations in illumination, pose, scale, instance, and material type, with KTHTIPS2b remaining especially challenging.
- Performance: CNN descriptors combined with IFV have broad advantages, but MRELBP and ScatNet achieve perfect accuracy on Outex TC10 and TC12 through rotation and grayscale invariance.
- Performance: CNN-based methods require high computational complexity and memory, while LBP-type methods offer more efficient feature extraction and traditional descriptors remain useful for real-time or degraded inputs.
- Performance: Performance is nearly saturated above 99.5% on Outex TC10, Outex TC12, CUReT, KTHTIPS, UIUC, UMD, and ALOT, whereas FMD, DTD, and KTHTIPS2b remain harder.
7 Discussion and Conclusion
The survey concludes that texture representation has advanced substantially but still faces demands from realistic data, efficient deployment, limited annotation, semantic understanding, and small-image analysis. It identifies large-scale datasets, robust and compact representations, data-efficient learning, and semantic attributes as priorities for future work.
- Discussion and conclusion: The survey organizes recent achievements, method categories, merits and demerits, benchmark datasets, representative performance, and promising directions for texture representation.
- Open challenges: Large-scale texture datasets are needed because complex variability in big image data exposes inadequacies of conventional handcrafted descriptors and motivates representation learning.
- Open challenges: Most descriptors remain insufficient for accurately and robustly discriminating massive texture categories across real-world scenes at a level comparable to human vision.
- Open challenges: Compact representations must address scalability, computation, storage, data demands, and power consumption, especially on embedded sensors, handheld devices, and mobile platforms.
- Open challenges: Texture learning should reduce dependence on annotated data for applications where labels are limited or expensive to collect.
- Open challenges: Future representations should move beyond low-level identification and categorization toward semantic, easily describable texture attributes for human-centric applications.
- Open challenges: New descriptors should be evaluated on small images because many applications require high-speed analysis for facial analysis, segmentation, defect detection, and tracking.