PaperlayerView exact revision on arXiv

Source-linked AI summary

Describing Textures in the Wild

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, Andrea Vedaldi

arXiv:1311.3618v2cs.CV

TL;DR

The paper asks whether a universal vocabulary can describe texture patterns and be estimated reliably from images. It builds the Describable Textures Dataset and applies Improved Fisher Vectors, finding strong performance for texture attributes and material recognition across benchmarks.

  • Problem

    The paper investigates whether universal semantic attributes can describe diverse texture patterns, be reliably estimated from images, and support useful analysis tasks.

  • Method

    The authors create a 5,640-image dataset with 47 annotated texture attributes and apply Improved Fisher Vectors to recognize them in natural images.

  • Results

    Improved Fisher Vectors outperform specialized texture representations for attribute recognition and material recognition, reaching 63.3% on FMD and 67.5% on KTH-TIPS2-b.

  • Takeaways & Limitations

    The learned describable attributes transfer across datasets and tasks, serving as useful predictors of materials and supporting fine-grained material distinctions.

  • Takeaways & Limitations

    The annotation procedure assumes annotators are deterministic and noise-free.

Abstract

from arXiv · show

Patterns and textures are defining characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this analytical dimension in image understanding, we address the challenging problem of describing textures with semantic attributes. We identify a rich vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected in the wild.The resulting Describable Textures Dataset (DTD) is the basis to seek for the best texture representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and show that, surprisingly, it outperforms specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that the describable attributes are excellent texture descriptors, transferring between datasets and tasks; in particular, combined with IFV, they significantly outperform the state-of-the-art by more than 8 percent on both FMD and KTHTIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.

1. Introduction

The paper introduces describable texture attributes and a dataset for recognizing and describing real-world texture patterns, then evaluates representations and applications across texture and material recognition.

  • Motivation: The paper targets automatic estimation of semantic texture properties, which had received less attention than object attributes despite their importance for describing patterned objects.The introduction frames textures such as patterns on shirts and butterfly wings as important descriptive properties while noting that texture attributes had been investigated only tangentially.
  • Dataset: 47 describable texture attributes and 5,640 jointly annotated texture images form the proposed dataset for representing diverse visual texture properties.The images are intended to support real-world applications and are inspired by large-scale visual datasets such as ImageNet and FMD.
  • Representation: The paper seeks a gold-standard texture representation that optimally recognizes describable attributes under challenging real-world conditions.This contribution is evaluated against specialized state-of-the-art texture representations and extends beyond the proposed dataset to material-recognition datasets.
  • Results: 63.3% accuracy on FMD and 67.5% on KTH-TIPS2-b show that the evaluated descriptors transfer effectively to material recognition.The introduction reports these results as evidence that the descriptors outperform specialized state-of-the-art texture representations across additional datasets.
  • Applications: The proposed describable attributes also support recognition and description when material is unimportant or known, and improve other texture-analysis tasks including material recognition.The introduction motivates applications to domains such as fabrics and wallpapers and reports affirmative experiments on FMD and KTH datasets.

2. The describable texture dataset

The Describable Textures Dataset (DTD) is a 5,640-image benchmark of real-world textures labeled with one or more attributes from a 47-word vocabulary. Its collection pipeline combines web-based image gathering, cost-efficient sequential annotation, and consensus-based handling of noisy labels.

  • The describable texture dataset: Each image is annotated with one or more adjectives selected from a vocabulary of 47 describable texture attributes.Examples include banded, cobwebbed, freckled, knitted, and zigzagged.
  • The describable texture dataset: Describable attributes can describe different materials and combine into richer multi-attribute descriptions, unlike assumptions that textures consist of a single material.For example, veined can describe leaves or marble, while marble can be veined, stratified, and cracked simultaneously.
  • The describable texture dataset: DTD contains 5,640 texture images collected from the web, with textures filling each image to avoid confounding texture description with segmentation.The dataset targets real-world applications in which recognizing texture properties is important.
  • DTD design and collection: The dataset construction selected 47 attributes, collected at least 120 representative images per attribute, added multiple labels per image, and addressed annotation noise.The collection process began with a large pool of images downloaded from Google and Flickr and filtered using Amazon Mechanical Turk.
  • Sequential join annotations: With roughly 10 annotations per image, the sequential procedure recovered all attributes for at least 75% of images and missed one out of four on average for another 20%.The method exploits attribute co-occurrence and image-specific classifier scores to prioritize which attributes receive annotations.
  • Handling noisy annotations: Five annotations from different annotators were combined into a consensus, while an annotator-quality model could remove or down-weight unreliable annotators.This addresses subjectivity and errors caused by distracted, adversarial, or unqualified annotators.

3. Texture representations

The section presents two texture representations for classifying perceptual attributes: the Improved Fisher Vector (IFV) and a compact 47-dimensional vector of DTD attribute-classifier responses. It details IFV’s construction and emphasizes its strong performance relative to specialized texture representations.

  • Texture representations: The system represents each texture image φ(ℓ) and uses an SVM score ⟨w, φ(ℓ)⟩ to recognize perceptual categories.The section frames texture recognition as representation followed by classification.
  • Improved Fisher Vector: IFV extracts dense, multiscale local SIFT descriptors, soft-quantizes them with a diagonal-covariance GMM, and aggregates first- and second-order statistics.Descriptors may first be decorrelated and optionally reduced by PCA.
  • Improved Fisher Vector: IFV applies a signed square-root transform to the stacked statistics and then performs l2 normalization.These operations produce the improved version of the Fisher Vector.
  • Improved Fisher Vector: IFV combines distinctive SIFT descriptors, expressive Fisher Vector pooling, and a square-root kernel map rather than conventional texture-analysis descriptors and histogramming.The section relates these design choices to prior texture-analysis ideas and object-recognition developments.
  • Improved Fisher Vector: The off-the-shelf standard IFV surpasses texture-specialized approaches, despite earlier work using SIFT or Fisher Vectors without testing the standard formulation.The authors describe this result as surprising and report the comparison in Sect. 4.1.
  • Attribute representation: The attribute representation collects DTD classifier responses into a compact 47-dimensional feature vector φ(ℓ) = (c1(ℓ), . . . , c47(ℓ)).These describable attributes are proposed as texture descriptors in their own right.

4. Experiments

Experiments show that IFV is a strong texture representation, outperforming BoVW on DTD and performing competitively or better across material-recognition benchmarks. DTD’s semantic attributes also transfer to material recognition and retrieval, while providing compact and interpretable texture descriptions.

  • Encoding comparisons on DTD: IFV and VLAD achieve about 60% mAP on DTD, significantly outperforming BoVW at 53.82% mAP.IFV uses 256 Gaussian modes, VLAD uses 512 K-means centers, and BoVW uses K-means encoding.
  • State-of-the-art material classification: On challenging material datasets, DTD attributes achieve 61.1% mean accuracy on KTH-TIPS2-b and 49.0% on FMD, surpassing aLDA at 44.6%.The attribute descriptor is 47-dimensional and is three orders of magnitude smaller than IFV, although IFVSIFT performs better.
  • Attribute transfer to Outex: 64.52% mAP is achieved on Outex retrieval when DTD attributes are retrained with IFV using LBPu local descriptors, exceeding the 49.82% mAP obtained with IFVSIFT.The result transfers from DTD-trained attributes despite Outex being a different data source; LBPu itself is not competitive on DTD.
  • Semantic interpretation: Attribute predictions show excellent semantic correlation with KTH-TIPS-2b categories, such as wrinkled aluminium foil and bumpy, pitted, porous, flecked bread.Example predictions report the top two attributes as very accurate and the third as correct in about half of cases.
  • Material descriptions: DTD attributes produce strong FMD classification for foliage, wood, and water, each exceeding 70%.The 47-dimensional descriptors extend the material-recognition results reported for FMD.
  • Material descriptions: Semantic attributes distinguish fine-grained materials, including knitted versus pleated fabrics, frilly versus pleated-and-veined foliage, and porous or pitted stones.They also separate swirly and spiralled water from bubbly and sprinkled water, while some glass images are identified as crystalline.

5. Summary

The paper introduces a large in-the-wild texture dataset labeled with 47 describable attributes and uses it to study semantic texture properties. It also ports the Improved Fisher Vector from object recognition to texture recognition.

  • Dataset: 5,640 images form an in-the-wild dataset jointly labeled with 47 describable texture attributes.The dataset supports studying semantic properties of textures and patterns in natural images.
  • Objective: The dataset targets extraction of semantic texture and pattern properties for real-world human-centric applications.
  • Representation: The study ports IFV, an object-recognition representation, to the texture domain to recognize describable attributes in natural images.
Loading 1311.3618v2…