PaperlayerView exact revision on arXiv

Source-linked AI summary

Visualizing and Understanding Deep Texture Representations

Tsung-Yu Lin, Subhransu Maji

arXiv:1511.05197v2cs.CV

TL;DR

Although CNN texture descriptors have advanced recognition, how they encode texture and categorical invariances remains unclear. This paper evaluates bilinear CNNs, probes their invariance and category pre-images, and reports favorable benchmark performance alongside attribute-based image manipulation.

  • Problem

    How CNN-based texture descriptors represent texture and categorical invariances remains unclear despite their use in recognition.

    Source: §1 · p. 1

  • Method

    The paper evaluates bilinear CNN texture features, investigates translational invariance, visualizes category pre-images, and explores attribute-based image manipulation.

    Source: §1 · p. 2

  • Results

    Bilinear CNNs compare favorably on texture and scene recognition, and translational invariance is useful for ImageNet object classification.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The models provide a parametric approach to texture synthesis and image-content manipulation using texture attributes.

    Source: §7 · p. 8

    Texture synthesis with the approach is computationally expensive, requiring hundreds of CNN evaluations and taking several minutes on a high-end GPU.

    Source: §5 · p. 6

Abstract

from arXiv · show

A number of recent approaches have used deep convolutional neural networks (CNNs) to build texture representations. Nevertheless, it is still unclear how these models represent texture and invariances to categorical variations. This work conducts a systematic evaluation of recent CNN-based texture descriptors for recognition and attempts to understand the nature of invariances captured by these representations. First we show that the recently proposed bilinear CNN model [25] is an excellent general-purpose texture descriptor and compares favorably to other CNN-based descriptors on various texture and scene recognition benchmarks. The model is translationally invariant and obtains better accuracy on the ImageNet dataset without requiring spatial jittering of data compared to corresponding models trained with spatial jittering. Based on recent work [13, 28] we propose a technique to visualize pre-images, providing a means for understanding categorical properties that are captured by these representations. Finally, we show preliminary results on how a unified parametric model of texture analysis and synthesis can be used for attribute-based image manipulation, e.g. to make an image more swirly, honeycombed, or knitted. The source code and additional visualizations are available at http://vis-www.cs.umass.edu/texture

1. Introduction

The introduction positions bilinear CNNs as texture descriptors, studies their invariance and category representations, and extends the model toward attribute-driven image editing. Related work situates these contributions among orderless CNN descriptors, texture synthesis, CNN visualization, and style editing.

1.1. Related work

  • Texture descriptors range from patch-statistic textons, bag-of-words, Fisher vectors, and VLAD to CNN-activation pooling, including Fisher-vector and bilinear representations.Source: §1 · p. 1, §1.1 · p. 2

1. Introduction

  • Bilinear CNN features perform favorably against Fisher-vector CNNs on texture datasets, with similar results reported for MIT Indoor scene classification.Source: §1 · p. 2
  • On ImageNet, bilinear CNNs trained from scratch achieve better accuracy without spatial jittering than corresponding fully connected CNNs trained with jittering.Source: §1 · p. 2
  • Category pre-images expose learned properties, while quilting initialization accelerates gradient inversion and global adjustment removes many quilting artifacts.Source: §1 · p. 2
  • A unified parametric texture model supports attribute-based edits, such as making images swirly or honeycombed and blending multiple texture attributes.Source: §1 · p. 2

1.1. Related work

  • Prior synthesis compares parametric filter-statistic methods with faster patch-based quilting, while CNN inversion [28] and example-driven style edits frame visualization and manipulation.Source: §1.1 · pp. 2–3

2. Methodology and overview

The framework uses orderless bilinear CNN features for texture recognition and synthesis, then inverts or optimizes images to visualize and manipulate texture attributes. It also evaluates recognition and invariance, inversion, and attribute-based manipulation.

2. Methodology and overview

  • Bilinear features average location-wise outer products of CNN activations, yielding an orderless representation suited to texture modeling.The framework uses these features for texture recognition, synthesis, inversion, and attribute-based manipulation.Source: §2 · pp. 3–4
  • Gatys et al. synthesize texture by optimizing an image to match bilinear features at multiple layers while regularizing it with a natural-image prior.The method uses squared feature loss and gradient descent from a random image; L-BFGS is one optimization option.Source: §2 · p. 3
  • Style-content synthesis combines texture reconstruction with content reconstruction, using features from a separate image to specify target content.The cited prior work preserves color and spatial structure when reconstructing from unpooled features.Source: §2 · p. 3
  • For attribute manipulation, the framework optimizes an image to match target attribute predictions while remaining similar in texture to the input.Losses for multiple target attributes can be blended with weights.Source: §2 · p. 3
  • The experiments use a 16-layer VGG network and 224×224 inputs for target features, with L-BFGS and per-loss gradient normalization during reconstruction.The input resize is primarily for speed; bilinear-feature size does not depend on image size, so other output sizes can be obtained by minimizing the objective.Source: §2 · p. 4

3. Texture recognition

The experiments compare bilinear CNN features with Fisher-vector CNN features on texture and scene recognition, and test scale, layer, and fine-tuning effects. Bilinear pooling performs comparably to Fisher-vector pooling, while scale and layer choices affect accuracy.

3. Texture recognition

  • Evaluation covers DTD, FMD, KTH-T2b, and MIT indoor, using dataset-specific protocols and mean accuracy across splits for DTD and FMD.KTH-T2b uses one sample for training and the remaining three for testing; DTD and FMD use ten random splits.Source: §3 · p. 4
  • B-CNN features use location-wise outer products with average pooling, while FV-CNN features use Fisher vectors from CNN filterbank responses.Both descriptors are signed-square-root and ℓ2 normalized before one-vs-all SVM classification.Source: §3 · p. 4

3.1. Experiments

  • Multiple input scales consistently improve accuracy over single-scale features.The evaluated scales are relative to a 224×224 image, with smaller-than-receptive-field and larger-than-1024×1024 scales excluded.Source: §3.1 · p. 4
  • Across input scales, B-CNN and FV-CNN performance is virtually identical on the evaluated texture and scene datasets.The authors conclude bilinear pooling is as good as Fisher-vector pooling for texture recognition; Table 1 reports mean per-class accuracy on DTD, FMD, KTH-T2b, and MIT indoor.Source: §3.1 · p. 4, Table 1 · p. 5
  • Higher CNN layers yield better B-CNN classification accuracy than lower layers.Table 2 reports results for different layers on the first split of each dataset, using 224×224 inputs.Source: §3.1 · p. 5, Table 2 · p. 5
  • Combining features from all layers raises DTD accuracy from 69.9% to 70.7% and MIT indoor accuracy from 72.8% to 74.9%.The authors suggest that different layers capture complementary information; the multi-layer results are small but significant improvements.Source: §3.1 · p. 5
  • Fine-tuning on MIT indoor raises accuracy from 72.8% to 73.8%, while fine-tuning on texture datasets yields insignificant improvements.The authors suggest the texture-dataset results may reflect their small size.Source: §3.1 · p. 5

4. The role of translational invariance

Bilinear CNNs use orderless pooling to provide translation-robust features, and on ImageNet they outperform a standard VGG-M model trained with substantially more spatial jittering. Adding jitter improves B-CNN accuracy further, but the largest augmentation level yields only a small additional gain over f5.

4. The role of translational invariance

  • B-CNN replaces the final two fully connected layers with a linear classifier and softmax over normalized relu5 bilinear features.Its evaluation speed is 80% of standard CNN speed, making overall training times comparable.Source: §4 · p. 5
  • Training compares flip-only f1 with f5 and f25, which add five and 25 translations, respectively.For each example, one jittered copy is randomly selected during stochastic training.Source: §4 · p. 5
  • 38.7% top1 error with B-CNN f1 beats VGG-M’s 39.6% with f25; B-CNN f25 reaches 37.1%.B-CNN improves by 1.6% from f1 to f25, while going from f5 to f25 improves it by less than 1%.Source: §4 · p. 5

5. Understanding texture representations

Inverting bilinear features reveals that they tolerate patch rearrangement while retaining local structure, and that category reconstructions depend on combining layers. Image quilting can initialize gradient-based synthesis and style transfer, accelerating convergence while gradient descent removes quilting artifacts.

5. Understanding texture representations

  • Invariant images can shuffle patches while preserving important local structure, and reasonable synthesized textures use higher and lower layers together.The illustrated invariance uses bilinear features from relu1_1 through relu5_1.Source: §5 · p. 6, Figure 3 · p. 6
  • Quilting initialization starts at a lower objective and reaches the random-start final objective faster, while gradient descent removes many quilting artifacts.The synthesis optimization is computationally expensive, requiring hundreds of CNN evaluations and several minutes on a high-end GPU; quilting is orders of magnitude faster.Source: §5 · p. 6, Figure 5 · p. 7
  • Quilting-based initialization also starts style transfer at a lower objective and converges faster than random initialization.The style-transfer inputs provide content measured at conv4_2 and style measured by bilinear features.Source: §5 · p. 6, Figure 6 · p. 7
  • Category inversions show multi-scale, multi-colored dots for DTD’s dotted class, while the MIT bookstore and laundromat appear as bookshelves and laundry machines.Water and wood inverses are described as representative, and orderless features produce sharp images unlike simple category averaging.Source: §5 · pp. 6–7
  • Lower layers preserve color and small-scale structure, combining layers improves reconstructions, and relu5_3 alone gives poor inverse images despite best recognition accuracy.Figure 4 adds relu3_3, relu4_3, and relu5_3 successively to relu2_2.Source: Figure 4 · p. 6, §5 · p. 7

6. Manipulating images with texture attributes

The framework edits texture or image content toward specified texture attributes, and can blend categories into hybrid textures. Content-based editing preserves overall image structure, while texture-based editing targets the attribute’s texture.

6. Manipulating images with texture attributes

  • Texture-attribute editing can make an image or its content more honeycombed or swirly, with content reconstruction preserving overall image structure.Texture-matching examples use all layers, while content examples use relu4_2 reconstruction.Source: §6 · p. 7, Figure 8 · p. 8
  • Weighted category targets produce blended textures, with an intermediate transition selected manually between the two endpoint categories.Figure 9 depicts the first category on the left, the second on the right, and the transition in the middle.Source: §6 · p. 8

7. Conclusion

The paper studies CNN texture representations through recognition and inversion, concluding that translation invariance benefits texture, scene, and ImageNet object recognition. It also presents parametric texture synthesis and attribute manipulation, with quilting initialization speeding convergence and reducing quilting artifacts.

7. Conclusion

  • The paper’s study examines CNN texture representations through recognition tasks and inversion-based analysis of their invariances.Figure 9 illustrates hybrid textures blended according to weights for the two categories.Source: §7 · p. 8, Figure 9 · p. 8
  • Translation invariance is useful for texture and scene recognition and for ImageNet object classification.The conclusion describes the models as rich parametric approaches for texture synthesis and image-content manipulation with texture attributes.Source: §7 · p. 8
  • Quilting initialization significantly speeds gradient-based inversion convergence and removes many structural artifacts introduced by quilting.The authors identify computational expense as the key challenge and describe patch-based and gradient-based methods as complementary.Source: §7 · p. 8
Loading 1511.05197v2…