PaperlayerView exact revision on arXiv

Source-linked AI summary

Hybrid CNN and Dictionary-Based Models for Scene Recognition and Domain Adaptation

Guo-Sen Xie, Xu-Yao Zhang, Shuicheng Yan, Cheng-Lin Liu

arXiv:1601.07977v1cs.CV

TL;DR

The paper addresses the limited use of local discriminative and structural information alongside transferable CNN features. It combines MLR, CFV, and fully connected CNN representations, achieving strong results across scene recognition and domain adaptation, including 82.24% on Indoor-67 and 100% domain-transfer accuracy in a reported setting.

  • Problem

    CNN features are powerful and transferable, but dictionary-based features retain local discriminative and structural information that is not fully represented by CNN features.

  • Method

    The method combines MLR from clustered local parts, multi-scale CFV from the last convolutional layer, and fully connected CNN representations.

  • Results

    The hybrid representation achieves strong performance across scene recognition and domain adaptation, including 82.24% on Indoor-67 and 100% domain-transfer accuracy in a reported setting.

  • Takeaways & Limitations

    MLR, CFV, and fully connected CNN features provide complementary information for the evaluated scene recognition and domain adaptation tasks.

Abstract

from arXiv · show

Convolutional neural network (CNN) has achieved state-of-the-art performance in many different visual tasks. Learned from a large-scale training dataset, CNN features are much more discriminative and accurate than the hand-crafted features. Moreover, CNN features are also transferable among different domains. On the other hand, traditional dictionarybased features (such as BoW and SPM) contain much more local discriminative and structural information, which is implicitly embedded in the images. To further improve the performance, in this paper, we propose to combine CNN with dictionarybased models for scene recognition and visual domain adaptation. Specifically, based on the well-tuned CNN models (e.g., AlexNet and VGG Net), two dictionary-based representations are further constructed, namely mid-level local representation (MLR) and convolutional Fisher vector representation (CFV). In MLR, an efficient two-stage clustering method, i.e., weighted spatial and feature space spectral clustering on the parts of a single image followed by clustering all representative parts of all images, is used to generate a class-mixture or a classspecific part dictionary. After that, the part dictionary is used to operate with the multi-scale image inputs for generating midlevel representation. In CFV, a multi-scale and scale-proportional GMM training strategy is utilized to generate Fisher vectors based on the last convolutional layer of CNN. By integrating the complementary information of MLR, CFV and the CNN features of the fully connected layer, the state-of-the-art performance can be achieved on scene recognition and domain adaptation problems. An interested finding is that our proposed hybrid representation (from VGG net trained on ImageNet) is also complementary with GoogLeNet and/or VGG-11 (trained on Place205) greatly.

I. INTRODUCTION

CNN features offer strong transferable representations, while dictionary-based features retain local discriminative and structural information. The paper motivates combining these complementary sources for scene recognition and domain adaptation.

  • CNN features provide strong representations for visual tasks and can transfer across domains.
  • Traditional dictionary-based features preserve local discriminative and structural information that CNN representations may not explicitly capture.
  • The paper targets limited use of local image information in CNN-based scene recognition and domain adaptation.
  • The proposed approach combines mid-level local representation, convolutional Fisher vectors, and fully connected CNN representations.
  • The resulting hybrid representation is reported as complementary across its components and effective for scene recognition and domain adaptation.
  • Fisher vectors are built from multi-scale convolutional activations using Gaussian mixture modeling and gradient encoding.

III. THE PROPOSED HYBRID METHOD

The paper constructs MLR, CFV, and FCR from trained CNN models and combines them into a hybrid representation for visual tasks.

  • The pipeline constructs MLR, CFV, and FCR representations from trained CNN models.
  • The section presents the complete construction pipeline before detailing each representation component.
  • The method treats the three representations as coordinated components of one hybrid visual representation.

A. The Whole System

The system combines multi-scale MLR, CFV, and fully connected CNN representations, then concatenates them for linear SVM classification. MLR learns local part dictionaries through clustering and codes CNN activations from densely sampled image regions.

  • A. The Whole System: The pipeline extracts multi-scale CFVs, single-scale FCRs, and local discriminative MLRs, concatenates them, and classifies the hybrid representation with a linear SVM.The hybrid representation is denoted H_i = [MLR_i, CFV_i, FCR1_i, FCR2_i].
  • A. The Whole System: Selective search generates constrained local proposals, while spatial and feature activations jointly define a similarity graph for spectral clustering.Proposal sizes are constrained to 60×60–160×160 pixels, with width-to-height ratios below 3.
  • A. The Whole System: The method retains representative boxes from the largest clusters, adds context padding, and forward-propagates them to form image prototypes.The top T clusters are selected by box count, with one box sampled from each selected cluster.
  • A. The Whole System: K-means clustering of prototypes produces either class-specific or class-mixture part dictionaries containing local discriminative and multi-scale information.The class-specific dictionary clusters prototypes within each class before concatenation; the class-mixture dictionary clusters all prototypes together.
  • A. The Whole System: MLR applies dictionary coding to densely sampled CNN activations from square regions at multiple scales, using LLC to preserve locality efficiently.The sampled square sizes are 128×128, 92×92, and 64×64, with a 32-pixel step size.

C. CFV: Convolutional Fisher Vector

CFV represents CNN features from the last convolutional layer with multi-scale Fisher vectors. Its scale-aware training and pooling retain information from different input resolutions.

  • C. CFV: Convolutional Fisher Vector: CFV constructs Fisher vectors from descriptors in the last convolutional layer of CNN using multiple image scales.The convolutional descriptors are used without ReLU throughout the paper.
  • C. CFV: Convolutional Fisher Vector: Scale-specific CFVs are L2-normalized and max-pooled, while GMM sampling allocates descriptors proportionally across scales.The resulting pooled representation receives power and L2 normalization to produce the final CFV.

D. FCR: CNN Features from Well-Tuned Networks

FCR uses fully connected activations from well-trained CNNs as transferable global image representations. The method extracts two fully connected layers and normalizes them before classification.

  • D. FCR: CNN Features from Well-Tuned Networks: FCR extracts activations from the penultimate and last fully connected layers of a well-trained CNN applied to resized input images.FCR1 and FCR2 use ReLU activations and can be obtained from CNNs with or without fine-tuning.
  • D. FCR: CNN Features from Well-Tuned Networks: The fully connected representations are L2-normalized before being fed into the SVM classifier.

IV. EXPERIMENTS

The experiments evaluate the hybrid representation in scene recognition and visual domain adaptation against traditional and CNN-based state-of-the-art models.

  • IV. EXPERIMENTS: The evaluation covers two applications: scene recognition and visual domain adaptation.The comparisons include both traditional models and CNN-based models.

A. Datasets and Experimental Settings

The experiments evaluate hybrid CNN representations across scene-recognition and domain-adaptation datasets, using multiple CNN architectures, domains, and fixed implementation settings. The evaluation includes MIT Indoor-67, SUN-397, and Office domain-transfer tasks.

  • CNN models: AlexNet and VGG-19 are trained on ImageNet, whereas GoogLeNet and VGG-11 are trained on Place205 to test complementary representations.
  • Datasets: MIT Indoor-67 contains 15,620 images across 67 indoor scenes with low inter-class variance, using approximately 80 training and 20 testing images per class.
  • Datasets: SUN-397 contains 397 indoor and outdoor scene categories, with 50 images per class used for training and 50 for testing.
  • Datasets: Office domain adaptation evaluates Amazon, Dslr, and Webcam domains, including Amazon→Webcam, Dslr→Webcam, and Webcam→Dslr transfers across five partitions.
  • Domain adaptation: Domain adaptation constructs hybrid representations for source and target data, trains the classifier on source representations, and tests it on target representations.
  • Implementation: The implementation fixes spectral-clustering settings, uses 64 Gaussian components for Fisher vectors, extracts specified whole-image or cropped inputs, and sets SVM C to 1.

B. MIT Indoor-67 Experiments

MIT Indoor-67 experiments analyze dictionary size, dictionary type, fine-tuning, and hybrid representation performance with AlexNet and VGG-19. The results favor class-specific dictionaries and show that the VGG-19 hybrid reaches 82.24%.

  • Dictionary analysis: The experiments vary part-dictionary size for class-mixture and class-specific dictionaries without fine-tuning the CNN.Dictionary size equals representative parts per category multiplied by the total number of categories.
  • Dictionary analysis: Class-specific dictionaries consistently outperform class-mixture dictionaries, and their K-means learning is faster.The authors identify the class-specific dictionary as the preferred second-stage clustering choice.
  • Hybrid representations: The hybrid representations that include MLR outperform counterpart representations without the part-based component across the evaluated dictionary sizes.
  • Fine-tuning: Fine-tuned AlexNet produces classification rates comparable to representations computed from AlexNet without fine-tuning.The later experiments therefore use CNN models without fine-tuning.
  • Final comparison: 82.24% is achieved by the VGG-19 hybrid representation on Indoor-67, reported as the best result based on an ImageNet-trained network.The MLR dictionary size is 67×30 for VGG-19 in this comparison.

C. SUN-397 Experiments

On SUN-397, the hybrid representation achieves the best reported results under both AlexNet and VGG-19 configurations. The class-specific dictionary outperforms the class-mixture counterpart, while enlarging the dictionary can further improve results.

  • The hybrid representation achieves the best reported SUN-397 results under both AlexNet and VGG-19 configurations.
  • 57.31% under AlexNet exceeds the 56.20% result from fine-tuning PlaceNet trained on a large place database.
  • The class-specific dictionary performs better than the class-mixture dictionary on SUN-397.
  • Increasing the fixed dictionary size beyond 3970 can produce better results.

D. Domain Adaptation Experiments

The hybrid representation is evaluated for unsupervised and semi-supervised domain adaptation across Amazon, Webcam, and Dslr domains. It substantially outperforms competing approaches in some transfers, while large domain shifts can reduce the value of local information.

  • The experiments cover unsupervised and semi-supervised adaptation across Amazon, Webcam, and Dslr domains, using AlexNet.
  • Using stronger CNN models or models trained with DDC and DAN can further enhance the representation.
  • The method performs much better than other state-of-the-art approaches for D →W and W →D transfers.
  • 100% domain-transfer accuracy is obtained from Webcam to Dslr when all Webcam images and three additional Dslr samples per class are used for training.
  • Large domain bias between Amazon and Webcam may reduce the importance of local information, yielding only comparative results for A →W.

E. Parameter Analysis

Parameter analysis shows that MLR is somewhat sensitive to its weighting parameters, whereas the final hybrid representation is robust across tested settings. The selected setting therefore remains effective for classifier training and prediction.

  • The hybrid representation reaches 72.94% at (λB, λF) = (0.7, 0.3), only slightly above 72.89% at (1, 0).
  • MLR is slightly sensitive to λB and λF, with its best result obtained when λF = 0.
  • The final hybrid representation is robust to λB and λF, reducing the need to test many parameter combinations in applications.
  • The hybrid representation using the selected MLR setting remains an effective feature for classifier training and prediction.

F. Complementarity with Other Nets

The paper combines MLR, CFV, and FCR with representations from other CNNs and reports complementary performance. Constructing the full representation requires summing the costs of its component stages.

  • The VGG-19 hybrid representation is combined with GoogLeNet and VGG-11 representations to test complementarity.
  • The combined representations achieve performance reported as much better than current state-of-the-art results.
  • Total construction time sums the costs of MLR, CFV, FCR, linear-SVM training, and testing.
  • Construction takes approximately two hours on SUN-397 and less than one hour on Indoor-67 and Office databases.
  • The hybrid representation combines MLR, CFV, and FCR, whose complementary information improves accuracy over traditional CNN features.
Loading 1601.07977v1…