PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic-Aware Scene Recognition

Alejandro López-Cifuentes, Marcos Escudero-Viñolo, Jesús Bescós, Álvaro García-Martín

arXiv:1909.02410v3cs.CV

TL;DR

Scene recognition remains challenging because classes can share objects, and CNN performance still lags other vision tasks. This paper uses semantic segmentation to gate RGB features through a two-branch attention network, reporting state-of-the-art results across four datasets with fewer parameters.

  • Problem

    Scene classes can share objects, and scene-recognition performance remains below that of other image-understanding tasks.

    Source: §1 · p. 1, §2.1 · p. 5

  • Method

    A two-branch CNN uses semantic-segmentation features with attention to gate RGB features and refocus classification toward scene-discriminative elements.

    Source: §3.1 · p. 8, §5 · p. 26

  • Results

    The method outperforms reported state-of-the-art methods on four public datasets while significantly reducing network parameters.

    Source: §5 · p. 27

  • Takeaways & Limitations

    The model simplification decreases training and inference times and the amount of data needed for learning.

    Source: §5 · p. 27

    Flawed or imprecise semantic segmentation can misdirect attention and lead to incorrect scene predictions.

    Source: §4.7 · pp. 25–26

Abstract

from arXiv · show

Scene recognition is currently one of the top-challenging research fields in computer vision. This may be due to the ambiguity between classes: images of several scene classes may share similar objects, which causes confusion among them. The problem is aggravated when images of a particular scene class are notably different. Convolutional Neural Networks (CNNs) have significantly boosted performance in scene recognition, albeit it is still far below from other recognition tasks (e.g., object or image recognition). In this paper, we describe a novel approach for scene recognition based on an end-to-end multi-modal CNN that combines image and context information by means of an attention module. Context information, in the shape of semantic segmentation, is used to gate features extracted from the RGB image by leveraging on information encoded in the semantic representation: the set of scene objects and stuff, and their relative locations. This gating process reinforces the learning of indicative scene content and enhances scene disambiguation by refocusing the receptive fields of the CNN towards them. Experimental results on four publicly available datasets show that the proposed approach outperforms every other state-of-the-art method while significantly reducing the number of network parameters. All the code and data used along this paper is available at https://github.com/vpulab/Semantic-Aware-Scene-Recognition

1. Motivation and Problem Statement

Scene recognition remains difficult because different classes can share similar objects, while greater network capacity has yielded only modest performance gains. The paper proposes using semantic segmentation to guide attention in a two-branch CNN, reporting strong results without increasing network size.

1. Motivation and Problem Statement

  • Scene classes can share objects and object distributions, creating blurry inter-class boundaries that make scene recognition difficult.The task must balance repeatable features for within-class variation with distinctive features that separate classes.Source: §1 · p. 2
  • Increasing CNN capacity has not produced proportional scene-recognition gains: DenseNet-161 and VGG-16 outperform their cited smaller counterparts by only 2.9% and 1.6%, respectively.The comparisons are against AlexNet for DenseNet-161 and GoogLeNet for VGG-16.Source: §1 · p. 2
  • The method achieves the best reported Places-365 Standard validation results using 67.13% fewer units than VGG-16 and 62.73% fewer layers than DenseNet-161.The approach uses semantic-driven attention without increasing the network’s width or depth.Source: §1 · p. 3
  • The proposed end-to-end two-branch CNN combines RGB and semantic information, using segmentation-derived attention to reinforce learning of relevant scene content.The stated aim is to improve scene recognition without increasing network capacity or depth.Source: §1 · p. 3

2. Related Works

Prior scene-recognition work spans handcrafted features, CNNs, semantic context, multimodal inputs, and attention. This paper combines RGB imagery with semantic segmentation through a CNN-based attention mechanism.

2.1. Scene Recognition

  • CNN-based scene-recognition methods outperform handcrafted features, while similar objects across classes remain a challenge and patch-based object descriptors omit spatial relations.The proposed approach uses semantic segmentation to exploit those spatial relationships and reports equal or better performance with fewer network units and hyperparameters.Source: §2.1 · pp. 4–5

2.2. Semantic Segmentation

  • Semantic segmentation assigns an object or stuff label to every pixel; motivated by contextual modeling and spatial relationships, the authors explore it as an added information modality.The paper contrasts this use of segmentation with scene-recognition methods based on object information.Source: §2.2 · p. 6

2.3. Multi-Modal Deep Learning Architectures

  • Multimodal models use complementary inputs such as depth and optical flow across vision tasks; this paper combines RGB images with semantic segmentation using CNN attention instead of traditional feature combinations.The cited traditional combinations include linear combination, feature concatenation, and averaging.Source: §2.3 · pp. 6–7

2.4. Attention Networks

  • Attention modules refocus CNNs on task-relevant content; here, a learned map based on semantic segmentation is intended to reinforce features from common scene objects and hinder non-specific ones.The paper motivates attention as a way to emphasize relevant image content during learning.Source: §2.4 · p. 7

3. Proposed Architecture

The proposed system combines RGB and semantic-segmentation branches, using semantic features to gate RGB features and focus scene recognition on learned relevant content. It trains the branches separately before freezing them to train the attention module and classifier.

3. Proposed Architecture

  • The two-branch CNN combines RGB and semantic-segmentation features through an attention module, then classifies the resulting feature tensor for scene recognition.The Semantic Branch supplies semantic guidance for gating RGB features.Source: §3 · pp. 7–8, §3.2 · p. 9

3.1. Motivations

  • Semantic representations train an attention map that gates RGB features, refocusing them toward scene-discriminative elements.The paper describes the resulting receptive field as enlarged and displaced toward elements learned by the Semantic Branch.Source: §3.1 · p. 8

3.2. Preliminaries

  • The network takes a normalized RGB image and its semantic-segmentation score tensor, and sends the two branches’ feature tensors to the Attention Module.The combined tensor is passed to a linear classifier for a K-class prediction.Source: §3.2 · pp. 8–9

3.3. RGB Branch

  • The RGB Branch uses ResNet-18 and forwards features from Basic Block 4 to the Attention Module.Basic Blocks 2, 3, and 4 implement residual connections.Source: §3.3 · p. 9

3.4. Semantic Branch

  • The Semantic Branch maps segmentation scores to semantic features; its output matches the size of a ResNet-18 feature map, and a shallow network is preferred because the input lacks texture information.The branch is designed to represent semantic labels and their spatial and channel inter-dependencies.Source: §3.4 · p. 9

3.5. Attention Module

  • The Attention Module derives semantic gating features and RGB features, multiplies them to produce semantic-weighted features, and forwards those features to a classifier.The classifier uses average pooling, dropout, and a fully connected layer before inferring scene posterior probabilities.Source: §3.5 · pp. 10–11

3.6. Training Procedure and Loss

  • Training first fits the RGB and Semantic Branches separately, then freezes them while training the Attention Module and linear classifier to avoid one branch dominating optimization.Both stages minimize a logistic function using Negative Log-Likelihood loss.Source: §3.6 · pp. 11–12

4. Experiments and Results

The experiments evaluate the proposed network on four public datasets, explain its implementation and metrics, and use ADE20K ablations to assess design choices before state-of-the-art comparisons.

4. Experiments and Results

  • The experiments evaluate the network on ADE20K, MIT Indoor 67, SUN 397, and Places365, with ablations on ADE20K and state-of-the-art comparisons on three datasets.The section presents a varied set of ablations to assess design decisions.Source: §4 · p. 12, §4.2 · pp. 14–15, §4.4 · p. 16

4.1. Implementation Design

  • Training uses Deep Frank-Wolfe with an initial learning rate of 0.1, momentum 0.9, and weight decay 0.0001; inference time increases 37.5% over the RGB baseline.The fixed semantic segmentation network is trained on ADE20K with 150 classes, and semantic score tensors retain each pixel’s top three probabilities.Source: §4.1 · pp. 12–14

4.2. Datasets

  • The four benchmarks differ in scope: ADE20K has 1,055 scene classes and an unbalanced composition, while Places365-Standard uses 365 classes and 1.8 million training images.MIT Indoor 67 has 67 indoor classes; SUN 397 covers 397 indoor and outdoor classes.Source: §4.2 · pp. 14–15

4.3. Evaluation Metrics

  • Top@1, Top@2, and Top@5 measure whether the ground-truth class appears among the highest-scored predictions, while MCA addresses validation-set class imbalance.MCA is based on per-class Top@1 accuracy.Source: §4.3 · p. 15

4.4. Ablation Studies

  • The ADE20K ablations examine Semantic Branch architectures, attention mechanisms and architectures, and the effects of multimodal fusion and attention relative to the RGB Branch.The study uses ADE20K to reduce the influence of semantic-segmentation quality on ablation results.Source: §4.4 · p. 16

Influence of the Semantic Branch Architecture

The Semantic Branch ablations find that shallower networks can approach deeper models, while pretraining and ChAM offer additional gains; the selected four-layer design trades performance against complexity.

Influence of the Semantic Branch Architecture

  • ImageNet-pretrained ResNet-18 improves over training from scratch by 3.00% in Top@1 and 21.6% in MCA, while ChAM raises MCA by 2.6%–5.4%.The reported ChAM gains add an average of 0.1 million units; pretraining’s gains are described as moderate for the non-RGB semantic domain.Source: Influence of the Semantic Branch… · p. 16
  • Training from scratch yields similar Top@k and MCA performance for ResNet-18 and shallower Semantic Branches, with a relative improvement of 0.49% over the four-layer configuration.Three- and four-convolutional-layer configurations perform similarly, but the three-layer design requires larger kernels and more parameters for the same output size.Source: Influence of the Semantic Branch… · p. 16
  • A four-convolutional-layer Semantic Branch was selected as a performance-complexity trade-off, using 78.3% fewer units than ResNet-18.Its Top@1 and MCA were lower than ResNet-18 by 3.61% and 14.86%, respectively.Source: Influence of the Semantic Branch… · p. 16

Influence of the Attention Module

The attention-module study tests both alternative attention mechanisms and alternative architectures for the selected mechanism, using ADE20K ablations.

Influence of the Attention Module

  • The study evaluates different attention mechanisms while holding a similar architecture, with two convolutional layers per branch.The specified kernels are 3 × 3 × 512 and 3 × 3 × 1024.Source: Influence of the Attention Module · p. 16
  • A second ablation explores alternative network architectures after selecting an attention mechanism.Both ablations concern the attention module’s design.Source: Influence of the Attention Module · p. 16
  • Figure 8 depicts the different attention-mechanism architectures used in the ablation study.The figure caption recommends viewing it in color.Source: Figure 8 · p. 17

Attention Mechanisms

Hadamard attention outperforms additive and concatenation mechanisms in the reported comparisons, and gating semantic attention over RGB features is the selected mechanism.

Attention Mechanisms

  • G-RGB-H improves over non-gated Hadamard attention by 4.31% in Top@1 and 2.35% in MCA.The authors attribute this to normalized semantic attention after sigmoid, which gates RGB features while maintaining their numerical range.Source: Attention Mechanisms · p. 17
  • Reversing the gate—using RGB features to gate semantic features—reduces performance by 9.59% in Top@1 and 25.51% in MCA, so G-RGB-H is selected.The passage associates the reversed-gate result with semantic features lacking texture and color information and RGB features lacking object labels and clear boundaries.Source: Attention Mechanisms · pp. 17–18
  • Additive attention is 4.94% lower in Top@1 and 6.46% lower in MCA than G-RGB-H; concatenation is 1.13% lower in Top@1 and 6.38% lower in MCA.The study reports that concatenation also increases linear-classifier complexity.Source: Attention Mechanisms · p. 17, Table 3 · p. 18

Architecture of the G-RGB-H Module

The G-RGB-H attention module performs best when convolutional layers adapt RGB and semantic features before gating. The authors select a two-layer design with 3 × 3 spatial kernels.

Architecture of the G-RGB-H Module

  • Removing adaptation layers reduces Top@1 by 2.73% and MCA by 10.37% versus the two-layer configuration.The authors attribute this to unadapted RGB and semantic features hindering learning in their two-stage procedure.Source: Architecture of the G-RGB-H Module · p. 18
  • Using three rather than two attention-module layers reduces Top@1 by 2.15% and MCA by 7.77%, suggesting over-fitting.The authors compare a deeper three-layer module with the two-layer configuration.Source: Architecture of the G-RGB-H Module · p. 18
  • Channel-increasing 1 × 1 kernels outperform no convolutional layers, but remain behind kernels that also capture spatial dimensions.The spatial-kernel comparison supports the selected 3 × 3 design.Source: Architecture of the G-RGB-H Module · p. 18
  • The selected G-RGB-H strategy uses two convolutional layers with 3 × 3 × 512 and 3 × 3 × 1024 kernels at each branch.Source: Architecture of the G-RGB-H Module · p. 18

Influence of Semantic Segmentation in Scene Recognition

On ADE20K, the RGB and semantic branches complement one another, while reducing the semantic class set lowers performance. The state-of-the-art comparison spans 19 approaches and three datasets.

Influence of Semantic Segmentation in Scene Recognition

4.5. State-of-the-art Comparison

  • The state-of-the-art comparison covers 19 approaches across MIT Indoor 67, SUN 397, and Places365.Reported results are extracted from the respective papers unless explicitly stated otherwise.Source: §4.5 · pp. 19–20

Evaluation on MIT Indoor 67 and SUN 397

On MIT Indoor 67 and SUN 397, the proposed method improves on many state-of-the-art approaches while using fewer parameters. The ResNet-50 variant outperforms all reported methods on both datasets while retaining substantially lower complexity.

Evaluation on MIT Indoor 67 and SUN 397

Evaluation on Places 365

On Places365, the proposed method surpasses the reported DenseNet-161 and VGG-19 results. Qualitative analyses show attention refocusing and dataset-dependent object emphasis, while flawed segmentation can lead to erroneous predictions.

Evaluation on Places 365

  • On Places365, Ours exceeds DenseNet-161 by 0.73% in Top@1 accuracy and VGG-19 by 2.29%, while maintaining relatively low complexity.Source: Evaluation on Places 365 · p. 24

4.6. Qualitative Interpretation of the Attention Module

  • After attention, CAMs refocus from the RGB branch's center bias toward scene-indicative concepts such as kitchen microwaves, chicken-farm animals, and bathroom mirrors.The authors describe this refocusing as useful for disambiguating similar classes and focusing on discriminative objects in conflicting scenes.Source: §4.6 · p. 24
  • The final CAM is not a direct multiplication of RGB and semantic CAMs; it generally enlarges the focus area, potentially increasing information for scene discrimination.The authors suggest the semantic branch's larger CAMs may reflect its focus on small discriminative objects or broad object-transition areas.Source: §4.6 · p. 24
  • Attention on MIT Indoor 67 concentrates on indoor and shared classes, while attention on SUN 397 is more evenly distributed across indoor, outdoor, and shared classes.On MIT Indoor 67, attention to outdoor classes decreases relative to the baseline; the authors say the attended classes depend on dataset scene types.Source: §4.6 · pp. 24–25, Figure 11 · p. 25

4.7. Limitations of the Proposed Method

  • The method may not overcome erroneous information when semantic segmentation is flawed or imprecise.Source: §4.7 · pp. 25–26
  • When segmentation omits a discriminative computer, a correctly segmented cabinet can drive an Office image to a Kitchen prediction; mislabeling a court as water prompts water-related predictions.Source: §4.7 · p. 26

5. Conclusions

The method combines RGB and semantic-segmentation context through attention; on four datasets, it outperforms reported state-of-the-art methods while using fewer network parameters.

  • A two-branch CNN combines RGB and semantic information through attention, using transformed semantic features to gate convolved RGB features.The top-performing described strategy softmax-transforms convolved semantic-branch features before gating the RGB branch.Source: §5 · p. 26
  • Across ADE20K, MIT Indoor 67, SUN 397, and Places 365, the method outperforms every reported state-of-the-art method while significantly reducing network parameters.Source: §5 · p. 27
  • The reduced model size decreases training and inference times and the amount of data needed for learning.Source: §5 · p. 27
Loading 1909.02410v3…