PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep filter banks for texture recognition, description, and segmentation

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Andrea Vedaldi

arXiv:1507.02620v2cs.CV

TL;DR

The paper addresses limited semantic descriptions and unrealistic evaluation settings for textures, and revisits texture representations using deep convolutional features. It introduces describable attributes and clutter benchmarks, finding that orderless pooling of convolutional features supports strong recognition across diverse visual domains.

  • Problem

    Texture research has largely focused on texture identification and material recognition, while common datasets often assume textures fill images rather than appearing in clutter.

  • Method

    The paper introduces a forty-seven-attribute describable texture dataset, OpenSurfaces-based clutter benchmarks, and CNN convolutional layers used with classical pooling encoders.

  • Results

    Orderless pooling of convolutional neural network features provides a versatile texture representation with state-of-the-art recognition across texture, material, object, and scene tasks.

  • Takeaways & Limitations

    Texture attributes support human-centered description and retrieval, while deep convolutional filter banks extend orderless texture representations beyond texture-specific applications.

  • Takeaways & Limitations

    Existing texture datasets can lack natural variation and clutter, limiting their representativeness for recognizing materials in natural images.

Abstract

from arXiv · show

Visual textures have played a key role in image understanding because they convey important semantics of images, and because texture representations that pool local image descriptors in an orderless manner have had a tremendous impact in diverse applications. In this paper we make several contributions to texture understanding. First, instead of focusing on texture instance and material category recognition, we propose a human-interpretable vocabulary of texture attributes to describe common texture patterns, complemented by a new describable texture dataset for benchmarking. Second, we look at the problem of recognizing materials and texture attributes in realistic imaging conditions, including when textures appear in clutter, developing corresponding benchmarks on top of the recently proposed OpenSurfaces dataset. Third, we revisit classic texture representations, including bag-of-visual-words and the Fisher vectors, in the context of deep learning and show that these have excellent efficiency and generalization properties if the convolutional layers of a deep model are used as filter banks. We obtain in this manner state-of-the-art performance in numerous datasets well beyond textures, an efficient method to apply deep features to image regions, as well as benefit in transferring features from one domain to another.

1 Introduction

The paper expands texture understanding beyond recognition by introducing describable attributes and realistic clutter benchmarks, while showing that convolutional features with classical pooling support broad visual recognition.

  • Background: Orderless local-feature aggregation remains influential across image-understanding domains, including bag-of-visual-words and Fisher-vector representations.The paper situates its technical contribution within this established family of texture descriptors.
  • Semantic texture description: The authors introduce forty-seven human-interpretable texture attributes and a dataset annotated with them for describing generic texture patterns.The attributes also support organizing texture collections and providing compact descriptors for material recognition.
  • Realistic texture recognition: They create OpenSurfaces-based benchmarks for recognizing materials and texture attributes when textures appear in the wild and in clutter.This addresses the mismatch between field-of-view-filling texture datasets and practical natural-image settings.
  • Deep filter banks: They revisit classical texture models by using CNN convolutional layers as local feature banks with traditional pooling encoders.The approach truncates CNNs before fully connected layers and combines learned local descriptors with established texture representations.
  • Deep filter banks: The resulting representations achieve state-of-the-art recognition across materials, texture attributes, object categorization, and scene classification.The paper also reports benefits for applying deep features to image regions and transferring features between domains.

2 Describing textures with attributes

The paper introduces human-interpretable texture attributes and the Describable Textures Dataset (DTD) to support automatic description of generic texture patterns. It also develops efficient annotation and benchmark procedures, and examines how attributes relate to materials.

  • Dataset and attributes: DTD represents generic texture patterns with a vocabulary of 47 human-interpretable attributes and a dataset of real-world annotated images.The attributes include terms such as banded, cobwebbed, freckled, knitted, and zigzagged.
  • Dataset and attributes: Unlike material recognition, texture description predicts combinations of subjective attributes that may apply across different materials.For example, veined can describe leaves or marble, while marble may combine veined, stratified, and cracked attributes.
  • Dataset and attributes: DTD contains 5,640 web-collected texture images whose textures fill each image, separating description from the segmentation problem.The dataset was designed for real-world texture-property recognition while avoiding the additional challenge of locating texture regions.
  • Dataset and attributes: The attribute vocabulary was reduced from 98 candidate words by screening visually ambiguous terms and merging similar words into 47 representatives.The selection process excluded words tied mainly to surface shape, haptics, or poorly defined subjectivity.
  • Annotation and benchmarks: Sequential joint annotation uses attribute co-occurrence to reduce labeling effort to about 10 annotations per image instead of 47.With roughly 10 annotations per image, the procedure recovered all attributes for at least 75% of images and missed one on average for another 20%.
  • Annotation and benchmarks: DTD defines separate key-attribute and joint multi-attribute prediction challenges, evaluated with accuracy and/or mean average precision.The key-attribute task uses one label per image, whereas DTD-J permits multiple applicable labels and is evaluated using mAP.
  • Attributes versus materials: On FMD, attributes and materials were correlated but weakly informative of each other, with average entropy reductions of 14% and 0.5%, respectively.The paper reports that combinations of attributes can predict materials more reliably than individual attributes or materials alone.

3 Recognizing textures in clutter

This section extends texture recognition to realistic scenes where material and attribute regions appear in clutter. It builds benchmarks from OpenSurfaces for region recognition and joint segmentation-recognition.

  • Motivation: The clutter benchmarks address recognition and segmentation of materials and describable texture attributes in complex natural scenes.They remove the assumption that each texture fills the entire image, matching applications where textured regions occur within broader scenes.
  • Dataset construction: OpenSurfaces provides the starting point, with 25,357 images containing high-quality texture or material segments and 58,928 materially annotated segments used experimentally.Segments may also include annotations for material, viewpoint, BRDF estimates, and object class.
  • Dataset construction: The attribute benchmark augments OpenSurfaces with describable attributes but evaluates only eleven attributes having at least 100 matching segments.The selected attributes include banded, blotchy, checkered, flecked, gauzy, grid, marbled, paisley, pleated, stratified, and wrinkled.
  • Benchmark tasks: The benchmark includes region recognition with known extents and segmentation-recognition where textured regions must be identified in the image.The tasks are instantiated for both materials and describable attributes.
  • Benchmark tasks: Known-region material recognition uses classification accuracy and mAP, while attribute recognition uses mAP because multiple attributes may apply.Segmentation-recognition is evaluated with per-pixel classification accuracy.
  • Benchmark tasks: For attribute segmentation, a pixel prediction is correct when it matches any ground-truth label, and accuracy is normalized over all labeled pixels.Per-class normalization is not used because it is ill-defined for this multi-label setting.

4 Texture representations

The paper reframes texture representations as learned or handcrafted filter banks followed by pooling encoders, connecting classical orderless models with CNN descriptors. This hybridization can improve efficiency, accuracy, and transfer across image-understanding tasks.

  • Texture representations: Texture representations pool locally extracted image statistics uniformly and orderlessly to reduce sensitivity to long-range appearance variation.Textures range from regular repetitions to stochastic patterns, motivating local aggregation rather than reliance on global spatial configuration.
  • CNN hybridization: Replacing fully connected pooling with texture-style pooling accelerates image-subregion computation, improves recognition accuracy, and reduces the need for CNN fine-tuning.The paper links these benefits to applications such as detection and segmentation and to reduced domain shift.
  • Representation decomposition: The paper decomposes representations into local descriptor extraction and descriptor pooling, enabling systematic combinations of filter banks and pooling encoders.Pooling encoders include bag-of-visual-words, soft-assignment variants, and higher-order Fisher-vector statistics.
  • Descriptor families: Classical local descriptors include filters, patches, local binary patterns, and SIFT, while CNN features are learned from data and often transfer across domains.The paper contrasts handcrafted descriptors with learned deep features and discusses their use as alternative filter banks.
  • Learned filter banks: CNN convolutional layers act as learned nonlinear filter banks whose spatially distributed channel responses provide local descriptors.A descriptor at one spatial location is formed by collecting all feature-channel responses there.

5 Plan of experiments and highlights

The experiments compare representations across texture, object, scene, and segmentation tasks. They report strong performance for Fisher pooling of CNN descriptors, including improved recognition, domain transfer, and practical segmentation efficiency.

  • Recognition findings: FV-SIFT outperforms specialized texture descriptors on many texture-recognition problems, with further gains when SIFT is replaced by CNN local descriptors.The comparison establishes a progression from orderless pooling of handcrafted descriptors to orderless pooling of learned convolutional descriptors.
  • Recognition findings: FV-CNN is often significantly superior to FC-CNN in texture, scene, and object recognition, especially for deeper CNN architectures.The paper partially attributes this pattern to lower overfitting and easier integration of multiple image scales.
  • Recognition findings: FV-CNN descriptors can be compressed to the dimensionality of FC-CNN descriptors while preserving accuracy.
  • Realistic texture recognition: CNN-based descriptors substantially outperform existing methods on texture recognition in the wild for both materials and attributes.FV pooling is sometimes slightly and sometimes substantially better than FC pooling, with the difference increasing in cluttered scenes.
  • Domain transfer: FV pooling compensates for the domain gap between CNN training on scene-recognition and object-recognition domains.
  • Semantic segmentation: FV-VGG-VD combined with region proposals and CRF post-processing surpasses the state of the art on MSRC material segmentation, whereas FC-CNN is too slow for this scenario.

6 Experiments on semantic recognition

This section evaluates which local descriptors and pooling encoders work best across texture, object, and scene recognition. Representations are named by encoder–descriptor pairs, such as FV-SIFT, BoVW-CNN, and FC-CNN.

  • Experimental scope: The experiments compare representations across texture attribute, texture material, object, and scene recognition to identify task-dependent strengths.
  • Representation notation: An X-Y representation label denotes pooling encoder X applied to local descriptor Y, with FC-CNN representing conventional CNN features pooled by a fully connected layer.Examples include FV-SIFT and BoVW-CNN.

6.1 Local image descriptors and encoders evaluation

The local-descriptor and encoder evaluation uses densely sampled SIFT and ImageNet-pretrained CNN convolutional features across diverse recognition benchmarks. It also specifies multiscale extraction, representation dimensionality, classifier settings, and evaluation protocols.

  • Local descriptors: The experiments center on densely sampled SIFT descriptors and deep convolutional features extracted from ImageNet-pretrained CNNs.Most experiments use VGG-M convolutional features because of its performance and computational cost.
  • Multiscale extraction: CNN-based local descriptors are extracted at multiple image scales, using rescaling factors 2^s for s from −3 through 1.5.Scales producing images larger than 1024^2 pixels are discarded for efficiency.
  • Representation dimensionality: For K visual words, BoVW and LLC have K dimensions, VLAD has KD, FV has 2KD, while the FC representation has 4096 dimensions in the evaluated CNNs.
  • Classifier setup: Unless otherwise specified, the experiments use a nonlinear SVM with C fixed at 1 after cross-validation found nearly identical performance across tested values.Class-score recalibration improves classification accuracy but does not improve mean average precision.
  • Datasets and measures: The evaluation spans DTD, OpenSurfaces, FMD, KTH-T2b, PASCAL VOC 2007, and MIT Indoor under standard dataset protocols.DTD contains 47 attribute classes with 120 images each, while FMD contains 1,000 images across ten material categories.
  • Datasets and measures: Table 2 reports DTD classification accuracy averaged over the dataset’s ten predefined splits and highlights SIFT and convolutional features as the best-performing descriptor families.

6.1.3 Local image descriptors and kernels comparison

This section compares local descriptors, kernels, and pooling encoders for texture recognition, finding that learned CNN descriptors and orderless Fisher-style pooling outperform classical alternatives and FC pooling across tasks.

  • Codebook size: 4096 visual words is the selected BoVW codebook size because performance increases with K and saturates around that value.The relative benefit of increasing K is larger for SIFT and CNN features.
  • Kernels: Exponential-χ2 is the strongest tested kernel, followed by additive-χ2, Hellinger, and linear kernels.The kernel comparison is conducted for 1-vs-all SVMs trained on BoVW representations.
  • Local image descriptors: CNN local descriptors significantly surpass SIFT and handcrafted texture descriptors on describable texture recognition.Dense SIFT reaches 52.3% with BoVW-exp-χ2, versus 44% for the best linear-filter-bank variant.
  • Pooling encoders: IFV and VLAD outperform BoVW and LLC in almost all tasks, while FV is better than VLAD with SIFT and comparable with CNN features.CNN features retain a lead over SIFT regardless of the pooling encoder.
  • Pooling encoders: FV-CNN performs similarly to or better than FC-CNN, with especially large gains on DTD, OS+R, and MIT Indoor.For MIT Indoor, FV-CNN improves over FC-CNN by 5% with VGG-M and 13% with VGG-VD.

6.1.5 CNN descriptor variants comparison

The experiments compare CNN variants, network depth, and Fisher-vector capacity. Deeper VGG features and FV pooling improve performance, while combining FC and FV provides only modest additional gains.

  • CNN variants: VGG-M outperforms AlexNet, while VGG-VD substantially outperforms VGG-M on texture and material datasets.On FMD, FC-AlexNet obtains 64.8%, FC-VGG-M 70.3%, and FC-VGG-VD 77.4%.
  • Pooling comparison: Switching from FC to FV pooling can improve performance more than switching to a deeper CNN.On DTD, FC-VGG-M to FC-VGG-VD improves by 7.1%, whereas FC-VGG-M to FV-VGG-M improves by 11.3%.
  • Feature combinations: Combining FV-CNN and FC-CNN improves accuracy by 1–2% for VGG-VD and up to 3–5% for VGG-M, while adding FV-SIFT provides no significant benefit.Adding FV-SIFT improves performance by at most 1% and can degrade it on MIT and FMD.
  • Feature depth: 32% and 48% are the maximum absolute accuracy improvements obtained by increasing feature depth for VGG-M and VGG-VD, respectively.Performance increases fastest through conv3 before tapering, and the deepest features perform best.
  • FV capacity: Up to 15% improvement on DTD and 6% on OS occurs when increasing FV-CNN Gaussian components to about 64.Increasing the number beyond roughly 64 components provides little or no further advantage.

6.1.6 FV pooling vs FC pooling

This section examines why FV pooling can outperform FC pooling, testing orderless modeling, domain-transfer overfitting, and multi-scale information as possible explanations.

  • Interpretations: FV pooling may suit texture modeling better because it aggregates local descriptors without preserving their order.The passage presents this as an explanation for texture recognition gains, not for the full object-recognition result.
  • Interpretations: FV pooling may reduce domain-transfer overfitting because a linear classifier on FV uses a simpler, smoother function than stacked FC layers.The passage links this hypothesis to source-domain specialization and limited target-domain training data.
  • Scale analysis: Single-scale FV-CNN still outperforms FC-CNN by up to 5.6% for VGG-M and 9.1% for VGG-VD on datasets including FMD, DTD, and MIT Indoor.Using a single scale reduces the FV-CNN advantage by up to 3.8 percentage points, and FC-CNN wins in some cases.
  • Scale analysis: Multi-scale FV pooling is significantly better than multi-scale FC pooling, although multi-scale FC is slightly better than single-scale FC in most cases.The comparison uses fully-convolutional, sliding-window FC representations to accommodate multiple scales.

6.1.7 Dimensionality reduction of the CNN descriptors

The section studies PCA reduction of CNN local descriptors before Fisher-vector pooling and analyzes how Fisher components can localize distinctive texture features.

  • PCA reduction: The pooled FV-CNN descriptor has dimensionality 2KD, where K is the number of Gaussian centers and D is the post-PCA local-descriptor dimension.The experiment varies D from 512 to 32 and reports PASCAL VOC 2007 mAP as a function of 2KD.
  • PCA reduction: The PCA experiment varies CNN descriptor dimensionality from 512 to 32 while jointly varying the number of GMM components.Figure 8 reports results for VGG-M and VGG-VD using total descriptor dimensionality on the horizontal axis.
  • PCA reduction: PCA-reduced CNN descriptors improve performance by 1–2% mAP when the dimensionality reduction is compensated by more GMM components.Reducing descriptors to 64 or 80 dimensions appears to give the best performance at similar total representation dimensionality.
  • Component interpretation: Ranking GMM components by SVM subvector norm identifies components whose assigned local descriptors are then spatially marked in test images.The visualization extracts descriptors at a single scale and highlights locations associated with top-ranked components.
  • Component interpretation: Unsupervised GMM components can consistently localize distinctive characteristics of different texture types.The strongest components tend to fire on characteristic features in indicative DTD examples.

6.2 Evaluating texture representations on different domains

The paper evaluates texture representations across texture, object, and scene domains, including controlled, wild, cluttered, and domain-shifted settings. FV-CNN generally outperforms FC-CNN and transfers effectively across datasets.

  • Evaluation scope: Orderless pooling is evaluated beyond textures, including image, region, object, scene, and domain-transfer recognition tasks.The section compares texture-specific and general-purpose descriptors across multiple benchmarks and imaging conditions.
  • Controlled material recognition: 3–6% absolute gains in KTH-T2b and 8–10% in ALOT distinguish FV-CNN from FC-CNN across evaluated CNN architectures.CNN descriptors also improve over SIFT by up to 11% absolute accuracy on these material-recognition datasets.
  • Texture recognition in the wild: FV-CNN is about 3% better than FC-CNN on FMD and about 8% better on DTD, while VGG-VD improves accuracy by more than 11% over AlexNet.FV-CNN also substantially outperforms prior descriptors on FMD, with FV-VGG-VD reported at +24.7%.
  • Feature complementarity: Combining FC-CNN and FV-CNN improves performance by about 3% across the board, whereas adding FV-SIFT provides no further improvement.This complementarity pattern differs from an earlier result where SIFT complemented FC-CNN on an AlexNet variant.
  • Texture recognition in clutter: 59.5% accuracy on OS+R for FV-VGG-VD exceeds FC-VGG-VD at 43.4%, demonstrating a larger FV advantage when regions vary in size and shape.The section also notes that CNN features can be reused across multiple regions, avoiding recomputation.
  • Object and scene recognition: FV-CNN descriptors remain competitive across object and scene benchmarks, including 85.2% versus 84.9% mAP on PASCAL VOC.On MSRC+R, the best model reaches 99.1% per-pixel classification, although ground-truth segments make that number incomparable with other reported results.
  • Domain transfer: 81.0% versus 67.6% accuracy for VGG-VD shows FV-CNN’s advantage over FC-CNN under ImageNet-to-MIT Indoor transfer.The paper reports that FV-CNN also removes the observed advantage of domain-specific CNN training in this setting.
  • Comparison with alternative pooling: 74.2% versus 68.8% on MIT Indoor shows FV-CNN outperforming the comparable VLAD-based method for VGG-M.The comparison is also favorable for VGG-VD and when using the same CAFFE CNN.

7 Experiments on semantic segmentation

The paper extends region recognition to automatic texture-region recognition and segmentation using proposal generation, region classification, voting, and optional Dense-CRF refinement. FV-CNN is effective and computationally advantageous for segmentation, but proposal front ends can limit results.

  • Segmentation framework: Automatic segmentation combines low-level region proposals with a learned classifier, then resolves multiple pixel labels through voting.This proposal-based design supports both FC-CNN and FV-CNN descriptors.
  • Region proposal methods: Crisp regions produce a single image partition, whereas MCG generates thousands of overlapping proposals requiring pixel-label resolution.Both proposal methods rely on low-level cues such as color or texture consistency.
  • Limitation: Front-end superpixel segmentation failures can hamper downstream segmentation results, motivating post-hoc refinement with pixel-level algorithms.The paper treats front-end segmentation as a pooling mechanism whose output can be refined.
  • Dense-CRF refinement: Dense-CRF post-processing converts superpixel classification scores into pixel-level unary terms to refine boundaries using contextual and low-level image information.The classifier scores are interpreted as negative energies for pixel labels.
  • Results: About 10% lower per-pixel accuracy occurs when classifying crisp regions on OS than when using OS+R, while MSRC reaches 87.0%, just above 86.5% published performance.The MSRC comparison does not use dataset-specific training or CRF refinement.
  • Results: 55.7% accuracy is obtained on OS with FV-CNN and MCG proposals, comparable to crisp-region results.FC-CNN with MCG is impractical because CNN features would need to be computed from scratch for every region.
  • Computational efficiency: FV-CNN becomes computationally advantageous for segmentation because FC-CNN requires feature recomputation for every region proposal.This cost difference is less significant when classifying whole images or fixed regions.

8 Applications of describable texture attributes

The paper uses the 47 DTD attributes as compact semantic descriptors for material recognition and qualitative visualization. Attribute-based representations perform strongly on FMD and KTH-T2b while also exposing interpretable material-texture associations.

  • Attribute descriptors: A 47-dimensional descriptor is formed from predicted DTD attribute scores and used to represent images in target material datasets.The attribute predictor is trained on DTD, then evaluated with selected texture representations and classifiers.
  • Material recognition: 70.8% on FMD and 74.6% on KTH-T2b are achieved using FV-CNN with VGG-VD, improving substantially over FV-SIFT.FV-SIFT obtains 49.2% on FMD and 64.7% on KTH-T2b.
  • Material recognition: 77.1% accuracy on KTH-T2b slightly exceeds 76.0% state of the art, while 72.17% on FMD exceeds 57.7% by about 15%.Using an RBF classifier instead of a linear classifier improves performance by 1–3% across FMD and KTH-T2b experiments.
  • Outex retrieval: 64.5% mAP on Outex with DTD attributes over LBPu descriptors confirms that the Outex advantage comes from the local descriptor rather than the attribute vocabulary alone.The FV-SIFT attribute representation reaches only 49.82% mAP, compared with 63.3% for the LBPu-based result of the referenced method.
  • Interpretability: DTD attributes show semantic correlations with KTH-T2b material categories, such as wrinkled for aluminum foil and bumpy, pitted, porous, and flecked for bread.The descriptions are intended to be interpretable and are derived from attribute classifier scores.
  • Search and visualization: Attribute classifiers are applied to 10,000 wallpapers and bedding sets to rank images by association with texture attributes.Scores are recalibrated on the target dataset and converted to probabilities for visualization.

9 Conclusions

The paper combines a large describable-texture dataset, cluttered texture benchmarks, and deep orderless pooling. Its central conclusion is that pooling CNN features without spatial ordering yields a versatile descriptor across texture, scene, and object benchmarks.

  • Contributions: The paper introduces a 5,640-image in-the-wild dataset jointly labeled with 47 describable texture attributes.The dataset supports semantic analysis of textures and patterns and human-centric applications.
  • Contributions: New OpenSurfaces-derived benchmarks evaluate material and texture-attribute recognition when textures appear in clutter.The benchmarks complement the describable texture dataset and extend evaluation beyond textures filling the entire image.
  • Main finding: Orderless pooling of CNN features is reported as a remarkably good texture descriptor that is also versatile for scene and object description.The conclusion states that this approach reaches new state-of-the-art performance on several benchmarks.
  • Qualitative interpretation: Figure 12 summarizes the most frequent top-scoring attributes for KTH-T2b material categories using the proposed 47-word vocabulary.Each category is represented by its whole-category description while a single image provides visualization.
  • Qualitative interpretation: Figure 13 presents bedding and wallpaper examples with their top three predicted attributes and normalized classification scores.The examples illustrate attribute-based search and visualization on a 10,000-image dataset.
Loading 1507.02620v2…