PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep TEN: Texture Encoding Network

Hang Zhang, Jia Xue, Kristin Dana

arXiv:1612.02844v1cs.CV

TL;DR

Existing material and texture recognition pipelines use separate feature, dictionary, encoding, and classification components, while standard CNNs do not naturally provide the required orderless representation. Deep-TEN integrates a differentiable Encoding Layer into CNNs to learn these components jointly, and it outperforms prior methods across several material and texture benchmarks, with joint training improving recognition.

  • Problem

    Existing methods use modular pipelines with fixed features and encoders, while standard CNNs preserve spatial ordering instead of directly modeling orderless feature distributions needed for texture recognition.

  • Method

    Deep-TEN integrates a differentiable Encoding Layer into CNNs to jointly learn convolutional features, an inherent dictionary, and residual encoding in a single network.

  • Results

    Deep-TEN outperforms state-of-the-art methods on MINC-2500, KTH, GTOS, and 4D-Light, while achieving compatible results on FMD and MIT-Indoor.

  • Takeaways & Limitations

    The Encoding Layer provides an orderless, fixed-length representation and makes learned convolutional features easier to transfer to material and texture recognition.

  • Takeaways & Limitations

    Joint training from scratch is constrained on STL-10 because its labeled training set is limited, so learning CNNs from scratch is not expected to work well.

Abstract

from arXiv · show

We propose a Deep Texture Encoding Network (Deep-TEN) with a novel Encoding Layer integrated on top of convolutional layers, which ports the entire dictionary learning and encoding pipeline into a single model. Current methods build from distinct components, using standard encoders with separate off-the-shelf features such as SIFT descriptors or pre-trained CNN features for material recognition. Our new approach provides an end-to-end learning framework, where the inherent visual vocabularies are learned directly from the loss function. The features, dictionaries and the encoding representation for the classifier are all learned simultaneously. The representation is orderless and therefore is particularly useful for material and texture recognition. The Encoding Layer generalizes robust residual encoders such as VLAD and Fisher Vectors, and has the property of discarding domain specific information which makes the learned convolutional features easier to transfer. Additionally, joint training using multiple datasets of varied sizes and class labels is supported resulting in increased recognition performance. The experimental results show superior performance as compared to state-of-the-art methods using gold-standard databases such as MINC-2500, Flickr Material Database, KTH-TIPS-2b, and two recent databases 4D-Light-Field-Material and GTOS. The source code for the complete system are publicly available.

1. Introduction

Material and texture recognition requires orderless representations of feature distributions, but prior pipelines separate feature extraction, dictionary learning, encoding, and classification. Deep-TEN integrates these components so they can be tuned jointly for the recognition task.

  • Limitations of Existing Methods: Classic and recent recognition methods stack self-contained components, leaving features and encoders fixed after construction.These pipelines separate feature extraction, dictionary learning, encoding, and classifier training.
  • Motivation: Material and texture recognition must capture orderless feature distributions rather than preserve the spatial arrangement emphasized by standard CNN classifiers.Histograms or distributions provide an orderless encoding suited to spatial repetition.
  • Proposed Approach: Deep-TEN introduces a learnable Encoding Layer that ports dictionary learning and residual encoding into a single CNN layer.The layer is trained with the rest of the network rather than as a separate preprocessing stage.
  • Proposed Approach: The Encoding Layer produces an orderless feature-distribution representation, generalizes VLAD and Fisher Vector encoders, and accepts arbitrary input sizes as a fixed-length output.Its differentiable design supports end-to-end training through backpropagation.
  • Contributions: Deep-TEN jointly learns feature extraction, the inherent dictionary, and the encoding representation, making convolutional features easier to transfer across domains.The authors report that these features are likely to carry less domain-specific information.

2. Learnable Residual Encoding Layer

The Encoding Layer integrates dictionary learning and residual encoding into a differentiable CNN layer, producing orderless fixed-length representations for material and texture recognition. It jointly learns encoding parameters and an inherent dictionary under supervision.

  • Learnable Residual Encoding Layer: The encoder aggregates residual information for each codeword and outputs a fixed-length representation independent of the number of input descriptors N.The representation concatenates aggregated residual vectors with assignment weights and applies L2-normalization.
  • Learnable Residual Encoding Layer: The Encoding Layer ports dictionary learning and residual encoding into a single CNN layer while learning encoding parameters and an inherent dictionary supervised by the loss.Gradients pass through assignment weights, allowing extracted convolutional features to benefit from the encoding representation.
  • Learnable Residual Encoding Layer: Soft assignment replaces nondifferentiable hard assignment, while learnable smoothing factors model descriptor distributions with cluster-specific scales.Hard assignment selects the nearest codeword; soft assignment addresses codeword ambiguity and differentiability.
  • Relation to Other Methods: The Encoding Layer generalizes VLAD and Fisher Vector while retaining an orderless representation suited to material and texture recognition.It differs from NetVLAD by preserving the connection between visual words and their assignments, enabling codewords to be learned from descriptor distributions.
  • Relation to Other Methods: With one codeword and a fixed zero center, the Encoding Layer simplifies to sum pooling and, after L2-normalization, behaves like average pooling.Unlike ordinary pooling, the Encoding Layer integrates dictionary learning and residual encoding into the CNN.

3. Deep Texture Encoding Network

Deep-TEN combines a CNN with the Encoding Layer to learn texture representations end to end. The architecture supports domain transfer, arbitrary input sizes, and joint learning across varied labeled datasets.

  • Deep Texture Encoding Network: Deep-TEN is the deep convolutional neural network obtained by integrating the Encoding Layer into an end-to-end CNN architecture.The Encoding Layer serves as the central representation module for the network.
  • Domain Transfer: The residual encoder suppresses frequently appearing visual descriptors, preserving a property associated with discarding domain-specific information.Frequently occurring descriptors tend to have small residuals and assignment-weighted aggregate values.
  • Domain Transfer: The proposed framework makes convolutional features likely to be domain-independent because the entire network is optimized together with the dictionary and encoding representation.The paper contrasts this with approaches where domain-specific information is carried by fixed dictionaries and encoders.
  • Multi-size Training: The Encoding Layer pools convolutional features into fixed-length representations, allowing Deep-TEN to accept arbitrary input sizes.This combines sliding-window convolutional feature extraction with fixed-length encoding output.
  • Joint Deep Encoding: Deep-TEN supports joint learning across labeled datasets with different domains, labeling strategies, and image sizes.The paper frames this as enabling different visual tasks and datasets to benefit from shared convolutional features.

4. Experimental Results

The experiments evaluate Deep-TEN across material, texture, scene, and object-recognition datasets, comparing modular encoding baselines, input-size training strategies, and joint encoding. Deep-TEN outperforms the state of the art on four material and texture datasets, while multi-size training helps in-the-wild datasets but can slightly reduce performance when training-data variety is limited.

  • Datasets: Deep-TEN is evaluated on five material and texture datasets plus MIT-Indoor and Caltech-101.The material and texture benchmarks are MINC-2500, FMD, GTOS, KTH-TIPS-2b, and 4D-Light.
  • Baseline comparisons: Deep-TEN and FV-CNN consistently outperform FV-SIFT, while Deep-TEN outperforms FV-CNN under matched settings.The comparison uses single input image sizes and reports top-1 test accuracy mean±std %.
  • Multi-size training: 4.9% performance boost from multi-size training and 7.4% improvement over the best baseline are reported for Deep-TEN on MIT-Indoor.The experiments use 352×352 and 320×320 input sizes during training, with 352×352 used for testing.
  • Multi-size training: Multi-size training improves in-the-wild datasets such as MINC-2500 and MIT-Indoor but slightly decreases performance on FMD and GTOS.The reported explanation is the lack of training-data variety for FMD and GTOS; multi-size training also accelerates convergence.
  • State-of-the-art comparison: Deep-TEN outperforms the state of the art on MINC-2500, KTH, GTOS, and 4D-Light.On small-scale FMD and MIT-Indoor subsets, its results are within 2% and 4%, respectively, of state-of-the-art approaches.
  • Joint encoding from scratch: Joint training uses shared convolutional layers with separate encoding layers for CIFAR-10 and STL-10, whose classification losses are summed.The setup accommodates their different image sizes and is motivated by limited labeled data for STL-10.

5. Conclusion

The Encoding Layer bridges classic visual recognition and CNNs, improving input-size flexibility and feature transfer while Deep-TEN achieves superior results across material datasets.

  • The Encoding Layer bridges dictionary learning and residual encoding with CNN architecture in a single layer.
  • It allows arbitrary input image sizes and is intended to make learned convolutional features easier to transfer.
  • Deep-TEN outperforms traditional off-the-shelf methods and achieves state-of-the-art results on MINC-2500, KTH, GTOS, and 4D-Lightfield.
  • 55 frame/sec training and 290 frame/sec inference are achieved for 50-layer Deep-TEN on four Titan X Maxwell GPUs.

A. Encoding Layer Implementations

The implementation differentiates the Encoding Layer with respect to inputs, codewords, and smoothing factors, while using standard L2-normalization outside the layer.

  • The appendix provides explicit gradients for the Encoding Layer input and parameters.
  • L2-normalization is applied outside the Encoding Layer as a standard component.
  • Input gradients are accumulated across k independent sub-encoders using the chain rule.
  • Each sub-encoder depends only on its corresponding codeword, so codeword gradients are computed through that sub-encoder.
  • The smoothing-factor gradient is derived analogously because each sub-encoder depends only on its corresponding smoothing factor.
  • Multiplying assignment-weight numerator and denominator by e^φi avoids overflow in practice.

B. Multi-size Training-from-Scratch

Training Deep-TEN from scratch on MINC-2500 had inferior recognition performance to using pre-trained ResNet-50, while multi-size training accelerated convergence.

  • Multi-size training significantly improves Deep-TEN's convergence speed when trained from scratch on MINC-2500.
  • From-scratch training has inferior recognition performance compared with employing pre-trained ResNet-50.
  • Classic pipelines extract hand-engineered features, learn dictionaries offline, and encode feature distributions into fixed-length representations for classification.
  • The validation error improves less than the training error because testing uses a single image size for simplicity.
Loading 1612.02844v1…