PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge Guided Disambiguation for Large-Scale Scene Classification with Multi-Resolution CNNs

Limin Wang, Sheng Guo, Weilin Huang, Yuanjun Xiong, Yu Qiao

arXiv:1610.01119v2cs.CV

TL;DR

Large-scale scene recognition must handle both large within-class variation and ambiguity between categories. The paper combines multi-resolution CNNs with confusion-matrix class merging and extra-network soft labels, achieving leading results across scene-recognition benchmarks and challenges.

  • Problem

    Large-scale scene datasets exhibit large intra-class variation and label ambiguity as scene categories incorporate multi-level visual information and increasingly overlap.

  • Method

    The paper combines coarse and fine multi-resolution CNNs with two knowledge-guided strategies: merging ambiguous classes and using extra-network soft labels during training.

  • Results

    The method achieved superior performance across five benchmarks, second place at the Places2 challenge, and first place at the LSUN challenge.

  • Takeaways & Limitations

    Multi-resolution representations and knowledge-guided supervision provide an effective framework for large-scale scene recognition.

Abstract

from arXiv · show

Convolutional Neural Networks (CNNs) have made remarkable progress on scene recognition, partially due to these recent large-scale scene datasets, such as the Places and Places2. Scene categories are often defined by multi-level information, including local objects, global layout, and background environment, thus leading to large intra-class variations. In addition, with the increasing number of scene categories, label ambiguity has become another crucial issue in large-scale classification. This paper focuses on large-scale scene recognition and makes two major contributions to tackle these issues. First, we propose a multi-resolution CNN architecture that captures visual content and structure at multiple levels. The multi-resolution CNNs are composed of coarse resolution CNNs and fine resolution CNNs, which are complementary to each other. Second, we design two knowledge guided disambiguation techniques to deal with the problem of label ambiguity. (i) We exploit the knowledge from the confusion matrix computed on validation data to merge ambiguous classes into a super category. (ii) We utilize the knowledge of extra networks to produce a soft label for each image. Then the super categories or soft labels are employed to guide CNN training on the Places2. We conduct extensive experiments on three large-scale image datasets (ImageNet, Places, and Places2), demonstrating the effectiveness of our approach. Furthermore, our method takes part in two major scene recognition challenges, and achieves the second place at the Places2 challenge in ILSVRC 2015, and the first place at the LSUN challenge in CVPR 2016. Finally, we directly test the learned representations on other scene benchmarks, and obtain the new state-of-the-art results on the MIT Indoor67 (86.7\%) and SUN397 (72.0\%). We release the code and models at~\url{https://github.com/wanglimin/MRCNN-Scene-Recognition}.

I. INTRODUCTION

Large-scale scene recognition is challenged by visual inconsistency within categories and ambiguity between categories. The paper addresses these issues with multi-resolution CNNs and knowledge-guided disambiguation strategies.

  • Large-scale scene datasets contain hundreds of classes and millions of images, creating visual inconsistency and label ambiguity.
  • Visual inconsistency reflects large variation among images in one category, such as kitchens ranging from whole rooms to people with food.
  • Label ambiguity arises when categories overlap visually, as with baseball field and stadium baseball, which share representative objects.
  • The proposed framework captures multi-level cues with coarse and fine resolutions while using knowledge-guided strategies to disambiguate similar categories.
  • Experiments span ImageNet, Places, and Places2, followed by evaluations in major scene-recognition challenges.

II. RELATED WORK

The paper distinguishes its approach from prior scene-recognition, deep-network, multi-scale, and knowledge-transfer studies by targeting large-scale scene-specific challenges. Its method combines multi-resolution architectures with cross-task soft supervision for ambiguity.

  • Scene recognition: Prior scene-recognition work did not address the large visual inconsistency and category ambiguity raised by much larger scene databases.
  • Deep networks for image recognition: Unlike general deep image-classification studies, this work targets scene categories defined by local objects, global arrangements, and interactions.
  • Multi-scale representation: The proposed architecture uses distinct input sizes and network structures, unlike prior multi-scale methods relying on one input size and architecture.
  • Knowledge transfer: The soft-code transfer crosses visual tasks, such as object recognition and scene recognition, to address label ambiguity.

III. MULTI-RESOLUTION CONVOLUTIONAL NEURAL NETWORKS

The multi-resolution CNN is motivated by the multi-scale structure of scenes and uses BN-Inception as an efficient base architecture. Its design aims to represent objects and layouts at different spatial scales.

  • Scenes contain semantic components arranged in spatial layouts that can be observed at multiple scales, from an office desk to an entire office.
  • The architecture combines coarse-resolution CNNs for large-scale visual structure with fine-resolution CNNs for smaller visual patterns.
  • The method uses batch-normalized Inception because it is efficient and naturally supports multi-scale processing through its inception modules.
  • The original BN-Inception transforms a 224 × 224 image through convolutional and pooling layers before global average pooling.

B. Two-resolution architectures

The two-resolution architecture uses separate coarse and fine CNNs whose receptive fields describe different-sized image regions. Their complementary predictions support scene understanding, while knowledge-guided training addresses category ambiguity.

  • Two-resolution design: The proposed CNNs consist of coarse-resolution components based on normal BN-Inception and fine-resolution components using a deeper architecture.
  • Two-resolution design: Coarse CNNs process 224 × 224 regions and emphasize global arrangements or larger objects, while fine CNNs use higher-resolution images and added layers.
  • Prediction fusion: The two CNNs describe different-sized receptive-field regions, so their prediction scores are complementary and can be averaged.
  • Extension: The architecture can extend beyond two resolutions, with finer-resolution CNNs providing richer capacity and multiple resolutions complementing one another.
  • Knowledge-guided disambiguation: Knowledge-guided disambiguation addresses label ambiguity using confusion-matrix knowledge to merge similar classes and extra-network knowledge to provide additional supervision.
  • Comparison and training: Unlike conventional multi-scale training, the method uses different image sizes and distinct network structures, while scale jittering remains complementary.

A. Knowledge from confusion matrix

The method uses validation-set confusion patterns to identify ambiguous scene classes and merge them into more consistent supercategories. A bottom-up clustering procedure repeatedly merges the most similar classes until a threshold is reached.

  • Similarity from confusion: A confusion matrix derived from Places2 validation predictions provides pairwise evidence of ambiguity between scene classes.The matrix records cross-class prediction errors, with larger confusion values indicating greater ambiguity.
  • Examples: Examples merge baseball field with stadium baseball and cubicle office with office cubicles into shared supercategories.These examples illustrate how visually ambiguous labels are consolidated before training or prediction.
  • Similarity from confusion: The class similarity measure symmetrizes confusion probabilities so that the resulting metric is suitable for category merging.Cij denotes the probability of classifying class i as class j, and the measure is designed to be symmetric.
  • Supercategory construction: A bottom-up clustering algorithm repeatedly merges the pair of categories with the largest similarity and updates the similarity matrix.The process continues until no similarity value exceeds the threshold τ.
  • Supercategory construction: Varying τ reorganizes the original 401 Places401 classes into 351, 372, or 386 supercategories.At test time, predictions over supercategories are transferred back to the original classes by equally dividing each supercategory probability among its subclasses.

B. Knowledge from extra networks

The second disambiguation strategy uses pretrained knowledge networks to generate image-specific soft labels that complement scene hard labels. A multi-task objective trains CNNs to predict both label types, incorporating visual concepts from the extra networks.

  • Motivation: Class-level merging treats every image in an ambiguous class identically, whereas soft labels can represent image-level differences within that class.The approach addresses intra-class variation by allowing soft labels to vary across individual images.
  • Soft-label construction: Pretrained CNNs trained on datasets such as ImageNet or Places provide visual knowledge for producing soft labels.These models encode concepts that can supplement the original scene annotations.
  • Soft-label construction: A soft label represents each image as a distribution over the pretrained model’s class space, such as common objects or scene categories.For example, dining-room content may receive high probabilities for objects such as dining tables and doors.
  • Joint training: The soft-label strategy captures shared visual elements between ambiguous classes while retaining differences among individual images.The authors state that this additional knowledge acts as an inductive bias and can reduce over-fitting.
  • Joint training: The training framework predicts original hard labels and knowledge-network soft labels simultaneously in a multi-task objective.The objective balances the scene-label term and soft-code term through λ, with K1 and K2 denoting their respective dimensions.

V. EXPERIMENTS

The experiments evaluate the framework across six scene or image benchmarks, including large-scale ImageNet, Places, and Places2 datasets and transfer benchmarks MIT Indoor67 and SUN397. Validation-set comparisons are used for datasets whose test labels are unavailable, with dataset-specific training schedules.

  • Benchmark scope: The experimental study covers ImageNet, Places, Places2, LSUN, MIT Indoor67, and SUN397.The paper first evaluates large-scale recognition, then tests learned representations on additional scene benchmarks and discusses failure examples.
  • Evaluation protocol: Validation sets are used for ImageNet, Places, and Places2 comparisons because their test-set ground-truth labels are unavailable.The reported large-scale results therefore compare methods on validation data.
  • Dataset scope: ImageNet-1k contains 1,000 object categories, about 1,300,000 training images, and evaluates predictions using top5 error.Its validation set contains 50,000 images and its test set contains 100,000 images.
  • Dataset scope: Places205 contains 205 scene categories with approximately 2,500,000 training images and 100 validation images per category.The dataset also includes 200 test images per category.
  • Dataset scope: Places2 contains more than 10 million images across more than 400 scene categories, with 5,000 to 30,000 training images per class.The Places2 challenge 2015 dataset contains 401 scene categories.
  • Implementation: Training schedules vary by dataset, with learning-rate decreases and stopping iterations adjusted to dataset size.The stated schedules include stopping at 750,000 iterations for ImageNet and Places205, 1,300,000 for Places401, and 600,000 for Places365-standard.

B. Evaluation on multi-resolution CNNs

The multi-resolution evaluation compares single-resolution BN-Inception models with a two-resolution fusion and extends the study to multiple input resolutions. Combining coarse and fine-resolution CNNs improves recognition because their captured information is complementary.

  • Two-resolution evaluation: The two-resolution CNN combines normal BN-Inception trained at 256 × 256 with deeper BN-Inception trained at 384 × 384 using equal fusion weights.The comparison is conducted on ImageNet-1k, Places205, Places401, and Places365 validation sets.
  • Two-resolution evaluation: Fine-resolution CNNs trained from 384 × 384 images outperform coarse-resolution CNNs trained from 256 × 256 images across the evaluated datasets.The paper attributes this to richer visual content and more meaningful local details in fine-resolution inputs.
  • Two-resolution evaluation: Arithmetic fusion of normal and deeper BN-Inception further boosts recognition performance on the evaluated datasets.The authors interpret the improvement as evidence that multi-level information from the two resolutions is strongly complementary.
  • Baseline comparison: The two-resolution CNN outperforms AlexNet and VGGNet-16 by a large margin on the compared datasets.The framework is modular and can be applied to existing network structures.
  • Multiple resolutions: The multi-resolution study trains CNNs at 128, 256, 384, and 512 resolutions on ImageNet-1k and Places365.The comparison uses BN-Inception-based architectures to examine multiple input resolutions.

C. Evaluation on knowledge guided disambiguation

The evaluation tests confusion-matrix merging and extra-network soft labels as disambiguation strategies, finding improvements across Places401 and Places365, with scene-network guidance and model fusion particularly effective.

  • Confusion-matrix knowledge: 372 super categories produced the lowest Top5 classification error, 17.3%, when merging Places401 categories using confusion-matrix thresholds.The threshold τ was fixed at 0.5 for subsequent experiments.
  • Confusion-matrix knowledge: Confusion-matrix guidance slightly improved normal BN-Inception performance despite using less category information, suggesting ambiguity can encourage over-fitting to subtle category differences.The paper relates this behavior to visually close categories such as baseball field and stadium baseball.
  • Extra-network knowledge: Extra networks provide soft labels in a multi-task framework, with an ImageNet-trained object network and a Places205-trained scene network guiding CNN training.The scene network's best setting was λ = 0.5 for both normal and deeper BN-Inception architectures.
  • Extra-network knowledge: Scene-network guidance outperformed object-network guidance on Places365, achieving 13.4% versus 14.1% Top5 classification error.The authors associate this difference with stronger correlation between Places scene classes and Places2 categories.
  • Model fusion: Model fusion improved complementary normal and knowledge-guided CNNs, reducing Places365 error from 14.3% to 12.9% for normal BN-Inception models.For deeper BN-Inception on Places401, fusion reduced Top5 error from 16.7% to 15.8%.

D. Results at the Places2 challenge 2015

At the Places2 challenge 2015, the combined multi-resolution and disambiguation system placed second, while subsequent inclusion of B2 yielded stronger validation performance than the winner's result.

  • Challenge result: The SIAT MMLAB solution placed second in the Places2 challenge 2015, trailing the winner by 0.5% in Top5 test error.The submitted combination was A0+A1+A2+A3+B0 and used single-scale cropping.
  • Post-challenge validation: 15.5% Top5 validation error from B0+B1+B2 surpassed the winner method's 15.7% result after the challenge.B2 was omitted from the original submission because of the challenge deadline.

E. Results at LSUN challenge 2016

The framework transferred effectively to LSUN: fine-resolution and knowledge-guided models improved validation performance, fused models achieved 91.8% accuracy, and the test submission won the challenge with 91.6%.

  • LSUN validation: Deeper BN-Inception networks trained at finer resolution achieved better LSUN validation results than normal-resolution networks, 90.5% versus 89.9%.Object- and scene-guided CNNs each added roughly 0.5% over non-guided CNNs.
  • LSUN validation: 91.8% top1 validation accuracy was obtained by fusing prediction scores from multiple networks.The models were fine-tuned from Places401 pre-trained representations.
  • LSUN challenge: 91.6% top1 test accuracy secured first place at the LSUN challenge 2016 and exceeded Google's comparable baseline by 0.4%.The result came from fusing all models.
  • Cross-dataset transfer: Transferred representations achieved 86.7% on MIT Indoor67 and 72.0% on SUN397 without fine-tuning on the target datasets.The paper reports these as the best results on both benchmarks at the time.

G. Failure case analysis

Failure cases show that label ambiguity remains in visually similar or spatially co-occurring scene categories, motivating multi-label treatment for some images. The broader framework targets this challenge alongside intra-class variation through multi-resolution representations and knowledge-guided disambiguation.

  • Similar outdoor appearances confuse supermarket, petstore, and toyshop in Places401.
  • Spatial co-occurrence creates ambiguity among downtown, building, and skyscraper images.
  • Bridge versus tower and conference room versus classroom remain difficult distinctions in LSUN failure cases.
  • These failures suggest that some scene images are better treated as multi-label classification problems than with a single category label.
  • The proposed framework combines multi-resolution CNNs with knowledge-guided disambiguation to address intra-class variation and label ambiguity.
  • The method achieved superior performance across five scene-recognition benchmarks and placed second in the Places2 challenge and first in the LSUN challenge.
Loading 1610.01119v2…