PaperlayerView exact revision on arXiv

Source-linked AI summary

Scene recognition with CNNs: objects, scales and dataset bias

Luis Herranz, Shuqiang Jiang, Xiangyang Li

arXiv:1801.06867v1cs.CV

TL;DR

The paper studies scale-induced dataset bias in multi-scale CNNs and the limited effectiveness of combining scene-centric Places with object-centric ImageNet knowledge. It analyzes how the two CNN types respond across scales, proposes scale-specific networks and hybrid combinations, and reports improved scene-recognition performance. The authors conclude that carefully matching CNNs to scale can make object and scene knowledge complementary.

  • Problem

    Multi-scale CNNs often use one fixed feature extractor across scales, while effectively combining the different scene-centric and object-centric knowledge in Places and ImageNet remains unresolved.

  • Method

    The paper analyzes CNN responses across scales and builds multi-scale architectures using scale-specific Places-CNNs, ImageNet-CNNs, and optional fine tuning.

  • Results

    Carefully chosen scale-specific combinations achieve state-of-the-art scene-recognition performance, with results reported as excellent and reaching human recognition performance in some configurations.

  • Takeaways & Limitations

    ImageNet and Places are complementary when combined after reducing scale-related dataset bias, rather than treated as interchangeable general-purpose feature extractors.

  • Takeaways & Limitations

    Fine tuning is difficult on very local patches because scene labels may supervise object or object-part content, and the number of patches makes practical use sparse.

Abstract

from arXiv · show

Since scenes are composed in part of objects, accurate recognition of scenes requires knowledge about both scenes and objects. In this paper we address two related problems: 1) scale induced dataset bias in multi-scale convolutional neural network (CNN) architectures, and 2) how to combine effectively scene-centric and object-centric knowledge (i.e. Places and ImageNet) in CNNs. An earlier attempt, Hybrid-CNN, showed that incorporating ImageNet did not help much. Here we propose an alternative method taking the scale into account, resulting in significant recognition gains. By analyzing the response of ImageNet-CNNs and Places-CNNs at different scales we find that both operate in different scale ranges, so using the same network for all the scales induces dataset bias resulting in limited performance. Thus, adapting the feature extractor to each particular scale (i.e. scale-specific CNNs) is crucial to improve recognition, since the objects in the scenes have their specific range of scales. Experimental results show that the recognition accuracy highly depends on the scale, and that simple yet carefully chosen multi-scale combinations of ImageNet-CNNs and Places-CNNs, can push the state-of-the-art recognition accuracy in SUN397 up to 66.26% (and even 70.17% with deeper architectures, comparable to human performance).

1. Introduction

Scene recognition benefits from both object-centric and scene-centric knowledge, but scale changes create dataset bias when fixed CNN extractors process multi-scale patches. The paper addresses this by assigning scale-specific networks and combining Places-CNN with ImageNet-CNN more carefully.

  • ImageNet-CNNs excel at object recognition, whereas Places-CNNs excel at scene recognition, but CNNs lack invariance to significant scaling.The wider scale range and greater object count in scenes make this limitation especially important for scene recognition.
  • Using one fixed CNN across all patch scales creates dataset bias because data properties vary with scale while the feature extractor remains unchanged.At sufficiently large scaling changes, the data can shift from scene-like to object-like.
  • The paper proposes multi-scale architectures with scale-specific networks to address scale-related bias and combine Places and ImageNet knowledge.The approach adapts the feature extractor to the scale range being processed.
  • ImageNet-CNN and Places-CNN have different optimal scale ranges, reflecting their object-centric and scene-centric training data.The paper evaluates hybrid architectures and fine tuning as strategies for exploiting these complementary ranges.

2. Objects and scenes

ImageNet and SUN397 differ in object scale and density: ImageNet usually contains one large object, while SUN397 contains many smaller objects. Object recognition experiments show that CNN performance depends strongly on scale and context.

  • ImageNet objects are mostly large and often fill the image, whereas SUN397 objects are much smaller, producing an obvious scale mismatch.Object sizes are normalized relative to an equivalent training crop.
  • ImageNet images usually contain one large object, while SUN397 scene images typically contain many small objects.The comparison focuses on the distribution of object annotations per scene image.
  • The object study uses 100 images per category from the 75 most frequent SUN397 object categories, with precautions against selecting objects that are too small.The data support SVM training and testing on scene-derived object examples.
  • Canonical object examples are centered, resized to fill the crop, and then scaled from 100% down to 10%, with masked and background-preserving variants.Scaling the background-preserving variant progressively shifts crop content from object toward scene.
  • Masked-object accuracy is generally optimal above 70–80% size, where ImageNet-CNN performs slightly better than Places-CNN.As objects become smaller, both models degrade similarly, indicating limited robustness to scale changes.
  • For objects with background, performance is worse than for isolated objects; at smaller scales, Places-CNN substantially outperforms ImageNet-CNN.The result is consistent with smaller crops containing more contextual and scene-like content.

3. Multi-scale architecture with scale-specific networks

The proposed architecture processes increasingly large image patches with parallel CNNs, selecting the most suitable feature extractor for each scale. This scale-specific design targets bias that prior multi-scale methods left unresolved.

  • The multi-scale architecture applies several CNNs in parallel to patches extracted from increasingly larger versions of each input image.A fully convolutional implementation accelerates patch extraction, while max pooling aggregates patch features into image features.
  • The architecture selects ImageNet-CNN, Places-CNN, or a fine-tuned model according to each scale instead of using one CNN everywhere.Each selected model is intended to match the data distribution at its assigned scale.
  • Fine tuning can further adapt each CNN to its scale range, but requires resizing images and extracting training patches for the target scale.It is optional in the proposed architecture.
  • Unlike prior approaches that emphasize VLAD or Fisher-vector pooling while keeping the CNN fixed, this method focuses on the local CNN model and scale-induced bias.The distinction is the principled use of different feature extractors across scales.
  • Hybrid-CNN combined Places and ImageNet training data without accounting for the different object scales present in the datasets.The paper presents scale-specific combination as an alternative to that data-level fusion strategy.

4. Experiments on scene recognition

The paper evaluates how scale, training dataset, and dataset bias relate in scene recognition, then assesses multi-scale architectures that combine these factors.

  • Experiments on scene data analyze the relationship between scale, training dataset, and dataset bias through scene-recognition performance.The section then evaluates multi-scale architecture combinations.

4.1. Datasets

The paper evaluates its architectures on three widely used scene-recognition benchmarks spanning natural, indoor, man-made, and mixed scenes, with varying category counts and training-set sizes.

  • 15 scenes contains 15 natural and indoor categories, with models trained on 100 images per category.
  • MIT Indoor 67 contains 67 indoor categories, with 80 training images per category.Its object-rich indoor scenes are described as more challenging but potentially more amenable to ImageNet-CNNs applied to patches.
  • SUN397 contains 397 indoor, man-made, and natural categories, with 50 training images per category.It is described as challenging because of its many categories, limited training data, and variability in objects and layout properties.
  • SUN397 is widely accepted as the reference benchmark for scene recognition, and the experiments use seven scales.

4.2. Single scale

Recognition accuracy varies substantially with input scale: Places-CNN performs best at global scales, whereas ImageNet-CNN is strongest at intermediate and local patch scales. Fine tuning partly equalizes these scale-dependent curves but yields limited overall improvement, while discriminability and redundancy analyses reinforce the scale-specific behavior.

  • Accuracy: Places-CNN performs best globally, while ImageNet-CNN peaks at intermediate patch scales and outperforms Places-CNN at most local scales.The same pattern appears across 15 scenes, MIT Indoor 67, and SUN397.
  • Accuracy: Around scale 0.35, performance separates into an ImageNet-CNN-dominated object range and a Places-CNN-dominated scene range.This split later motivates the design of spliced architectures.
  • Accuracy: The proposed architecture combines scale-specific Places-CNNs at global scales with ImageNet-CNNs at local scales, pooling within each scale before concatenation.The spliced design uses a single splicing point to combine complementary scale ranges.
  • Effect of fine tuning: Fine tuning moderately improves ranges where the original CNN performs poorly, but produces marginal or no gains where its scale performance is already strong.Its overall improvement remains limited, although fine-tuned Places-CNN becomes best across the evaluated scale range on MIT Indoor 67.
  • Effect of fine tuning: Fine tuning mainly affects the top layers, resembling the effect of adding a BOW pooling layer, while intermediate layers remain biased toward the pretraining data.This limits the extent to which fine tuning can remove scale-related bias.
  • Discriminability and redundancy: ImageNet-CNN discriminability follows its accuracy curve, Places-CNN is most discriminative at global scales, and excessively local scales increase feature redundancy and noise.These information-theoretic measures broadly mirror the scale-dependent accuracy patterns, although discriminability does not always predict higher accuracy.

4.3. Two scales

Dual architectures combine two CNNs at different scales, with the strongest results pairing global Places-CNN features and local ImageNet-CNN features. This combination reaches 64.10% accuracy on SUN397 using two AlexNet CNNs.

  • 4.3. Two scales: 64.10% accuracy on SUN397 is achieved by a dual architecture using global Places-CNN features and local ImageNet-CNN features.The architecture concatenates two 4096-dimensional fc7 activations into an 8192-dimensional feature before SVM classification.
  • 4.3. Two scales: The best dual architectures are hybrid combinations rather than pairs using only one CNN type.Places-CNN operates at typically global 227x227 scales, while ImageNet-CNN extracts features from more local patches.
  • 4.3. Two scales: Combining ImageNet and Places after adapting each network to suitable scales is more effective than the earlier Hybrid-CNN approach.The dual architecture keeps object and scene knowledge separate while matching models to scales with similar properties.

4.4. Multiple scales

Multiple-scale experiments show that adding more networks is not automatically beneficial: scale-specific hybrid designs outperform fixed-model combinations, while excessive same-model scaling adds noise and extraction cost.

  • 4.4. Multiple scales: 78.21% accuracy is achieved on MIT Indoor 67 when all seven Places-CNN scale-specific features are concatenated after PCA reduction.The corresponding all-scales ImageNet-CNN architecture reaches 74.33%.
  • 4.4. Multiple scales: Adding more scales while keeping the same CNN model is marginally helpful but significantly increases extraction cost and representation noise.This limits the usefulness of indiscriminately expanding same-model multi-scale architectures.
  • 4.4. Multiple scales: 80.97% and 65.38% accuracy are obtained on SUN397 by a full hybrid architecture using Places-CNN at global scales and ImageNet-CNN at local scales.The architecture has one free parameter: the splicing point between the two CNN families.
  • 4.4. Multiple scales: The full hybrid architecture performs significantly better than a seven-network architecture using one fixed CNN model.Its scale assignment follows complementary optimal ranges for Places-CNN and ImageNet-CNN.
  • 4.4. Multiple scales: 66.26% accuracy on SUN397 is reached by combining full ImageNet-CNN and Places-CNN architectures in a 14-CNN model.This combination does not help on MIT Indoor 67 and helps only slightly on SUN397.

4.5. Deeper networks and other works

Deeper VGG architectures improve recognition overall, but their behavior differs from AlexNet: small-patch features are noisy, and larger combinations can underperform simpler baselines. The best reported accuracies reach 95.18% on 15 scenes, 86.04% on MIT Indoor 67, and 70.17% on SUN397.

  • 4.5. Deeper networks and other works: VGG experiments are consistent with AlexNet dual architectures but provide a more moderate gain.The deeper architecture does not improve every multi-network configuration.
  • 4.5. Deeper networks and other works: VGG applied to small patches tends to be noisy and performs poorly, while an intermediate three-scale hybrid performs slightly better than dual architectures.Combining more VGG networks was surprisingly disappointing and could perform worse than single-network baselines.
  • 4.5. Deeper networks and other works: 95.18% accuracy is the best result on the 15 scenes dataset, compared with 94.51% using AlexNet.The paper describes performance on this small dataset as somewhat saturated.
  • 4.5. Deeper networks and other works: 86.04% accuracy is achieved on MIT Indoor 67 and 70.17% on SUN397 with deeper architectures, compared with 80.97% and 66.26% using AlexNet.The SUN397 result is close to reported human expert performance and exceeds reported good-worker performance.

5. Conclusions

The paper interprets multi-scale scene recognition as requiring scale-aware feature extractors rather than a single generic CNN. It argues that properly combining scene-centric Places and object-centric ImageNet representations can achieve excellent recognition with only a few carefully chosen networks.

  • 5. Conclusions: Scaling images induces dataset bias between training and test data, significantly affecting recognition performance.The paper identifies this bias as arising from changes in feature distributions across scales.
  • 5. Conclusions: ImageNet-CNN and Places-CNN are implicitly tuned for different scale ranges corresponding to object scales and scene scales.This distinction motivates assigning different CNNs to different scales.
  • 5. Conclusions: Scale-specific networks are proposed as a principled way to address scale-related dataset bias and combine scene-centric and object-centric knowledge.The approach focuses on the local CNN model rather than the pooling method.
  • 5. Conclusions: Carefully combining global and local multi-scale features can reach human recognition performance with only two or three networks.The paper presents holistic Places-CNN recognition and local object recognition as compatible cases within a broader multi-scale framework.
Loading 1801.06867v1…