PaperlayerView exact revision on arXiv

Source-linked AI summary

Hybrid Knowledge Routed Modules for Large-scale Object Detection

Chenhan Jiang, Hang Xu, Xiangdan Liang, Liang Lin

arXiv:1810.12681v1cs.CV

TL;DR

Large-scale object detection often treats regions separately, leaving semantic correlations underused amid long-tailed classes and confusing categories. HKRM combines explicit and implicit commonsense knowledge through adaptive region-to-region reasoning. It outperforms Faster RCNN on large-scale benchmarks, with around 34.5% mAP improvement reported.

  • Problem

    Region-wise detection overlooks semantic correlations between objects and struggles in large-scale settings with rare classes and confusing categories.

    Source: §1 · p. 1

  • Method

    HKRM combines explicit linguistic knowledge and implicit commonsense constraints in adaptive region-to-region modules for global semantic reasoning.

    Source: §1 · p. 2

  • Results

    HKRM outperforms Faster RCNN [43] on ADE and VG (1000 or 3000 categories), with around 34.5% mAP improvement reported.

    Source: §1 · p. 2

  • Takeaways & Limitations

    HKRM shares visual features across related regions and is intended to alleviate difficult-region and category-imbalance problems.

    Source: §1 · p. 2

Abstract

from arXiv · show

The dominant object detection approaches treat the recognition of each region separately and overlook crucial semantic correlations between objects in one scene. This paradigm leads to substantial performance drop when facing heavy long-tail problems, where very few samples are available for rare classes and plenty of confusing categories exists. We exploit diverse human commonsense knowledge for reasoning over large-scale object categories and reaching semantic coherency within one image. Particularly, we present Hybrid Knowledge Routed Modules (HKRM) that incorporates the reasoning routed by two kinds of knowledge forms: an explicit knowledge module for structured constraints that are summarized with linguistic knowledge (e.g. shared attributes, relationships) about concepts; and an implicit knowledge module that depicts some implicit constraints (e.g. common spatial layouts). By functioning over a region-to-region graph, both modules can be individualized and adapted to coordinate with visual patterns in each image, guided by specific knowledge forms. HKRM are light-weight, general-purpose and extensible by easily incorporating multiple knowledge to endow any detection networks the ability of global semantic reasoning. Experiments on large-scale object detection benchmarks show HKRM obtains around 34.5% improvement on VisualGenome (1000 categories) and 30.4% on ADE in terms of mAP. Codes and trained model can be found in https://github.com/chanyn/HKRM.

1 Introduction

Region-by-region detection struggles with rare classes and scenes containing occlusion, ambiguity, and tiny objects. HKRM adds explicit and implicit commonsense reasoning through adaptive region-to-region connections, and outperforms Faster RCNN on large-scale benchmarks.

1 Introduction

  • The method targets region-based detection's reliance on sufficient per-category training data, a weakness under long-tailed class frequencies and increasingly difficult scenes.The cited challenges include heavy occlusion, class ambiguity, and tiny-size objects.Source: §1 · p. 1
  • HKRM combines explicit linguistic constraints, such as attributes and relationships, with implicit commonsense constraints, such as spatial layouts.The paper showcases attribute, pairwise relationship, and spatial knowledge forms.Source: §1 · p. 2
  • Its knowledge modules adapt region-to-region connections to each image rather than fixing category-level edges, and can be combined for hybrid reasoning.The overview describes knowledge-supervised adaptive edges and feature enhancement by individual modules.Source: §1 · p. 2, Figure 2 · p. 3
  • HKRM shares visual features across regions with related attributes or relationships, or spatial relationships, to address difficult regions and alleviate category imbalance.The paper specifically links this sharing to heavy occlusion, class ambiguity, and tiny-size problems.Source: §1 · p. 2
  • HKRM outperforms Faster RCNN [43] on ADE and VG (1000 or 3000 categories), with reported improvements of 26.5% on VG (3000 categories) and 30.4% on ADE.The introduction also reports around 34.5% mAP improvement, without specifying the benchmark in the supplied passage.Source: §1 · pp. 2–3

2 Related Work

Prior work addresses object detection, visual reasoning, and few-shot recognition, but commonly reasons over individual proposals or uses fixed graphs. HKRM instead uses adaptive region-to-region edges guided by hybrid knowledge forms.

Object Detection

  • Many detection methods operate on each proposal individually without reasoning, while commonly addressing only a small category set such as PASCAL VOC's 20 or COCO's 80.The passage contrasts those benchmarks with large-scale detection settings.Source: §2 · p. 3

Visual Reasoning

  • Visual reasoning work uses attributes, relations, graphs, or sequential models, but typically considers a fixed graph unlike HKRM's adaptive region-to-region edges.The cited passage also notes earlier relation methods using post-processing and graph structures.Source: §2 · p. 3

Few-shot Recognition

  • Few-shot recognition transfers information from few examples, including through attributes or knowledge graphs, whereas HKRM is explicitly routed and guided by hybrid knowledge forms.The passage contrasts HKRM with an implicitly learned knowledge-graph GNN.Source: §2 · p. 3

3 The Proposed Approach

HKRM adds explicit and implicit knowledge modules that build adaptive region-to-region graphs to enhance proposal features for detection. Explicit graphs use linguistic prior knowledge, while implicit graphs capture constraints not explicitly defined.

3.1 Overview

  • Each module builds an adaptive graph over proposal regions, enhances their features, and feeds the module outputs to the detection layers.The overview describes concatenating the enhanced features with the original features before producing final detection results.Source: §3.1 · pp. 3–4, Figure 3 · p. 4

3.2 Explicit Knowledge Module

  • Explicit knowledge is linguistically summarized commonsense, such as shared attributes and pairwise relationships, used to update region-graph edges.The edge updates are supervised by a mapping from a class-to-class knowledge graph.Source: §3.2 · p. 4

3.2.1 Module Definition

  • A stacked MLP learns each region-pair edge from visual features, with predicted edges trained toward prior-graph weights for the regions’ ground-truth classes.After row normalization, weighted graph propagation enhances region features; the evolved features are concatenated with the original features.Source: §3.2.1 · pp. 4–5

3.2.2 Module Specification with Different Knowledge

  • Attribute graphs encode class similarities from shared attributes, while relationship graphs encode semantic links such as spatial, subject-verb-object, and co-occurrence relations.Attribute edges use Jensen–Shannon divergence between class attribute distributions; relationship edges use frequency statistics and are symmetrized and row-normalized.Source: §3.2.2 · p. 5

3.3 Implicit Knowledge Module

  • The implicit module uses region features to learn multiple graphs for constraints such as spatial layouts, averages their edges, and adds identity connections.For spatial-layout inputs, region geometry is represented using relative bounding-box features and initial foreground probability.Source: §3.3 · pp. 5–6

4 Experiments

Experiments evaluate HKRM on large-scale detection benchmarks and compare it with baselines and knowledge-enhanced methods. Results include improvements across datasets, ablations of knowledge modules, cross-dataset transfer, and visual analyses of learned features.

4 Experiments

  • The evaluation covers VG1000 and VG3000, plus ADE targets among 445 classes overlapping with VG; ADE segmentation masks are converted to bounding boxes.VG settings use the 1,000 or 3,000 most frequent classes; evaluation includes mAP across IoU thresholds and scales.Source: §4 · p. 6

4.1 Comparison with state-of-the-art

  • HKRM reaches 7.8% overall AP versus Faster RCNN’s 5.8% on VG1000, 4.3% versus 3.4% on VG3000, and 10.3% versus 7.9% on ADE.The paper also reports about 1.5% average mAP improvement for the top 150 infrequent categories.Source: §4.1 · pp. 7–8
  • On MSCOCO, HKRM raises mAP from 34.9% to 37.8%; on PASCAL VOC, it performs 1.1% better than Faster RCNN and outperforms Spatial Memory Network.The three-module model adds about 2% parameters.Source: Table 2 · p. 7, §4.1 · p. 8
  • HKRM detects examples missed or challenged by Faster RCNN, including distant palm trees, overlapping small objects, and the rare category “pepperoni.”The comparison is qualitative and uses VG1000.Source: §4.1 · p. 8, Figure 4 · p. 8

4.2 Ablation Studies

  • Attribute knowledge alone raises overall AP over baseline by 1.6% on VG1000, 0.6% on VG3000, and 1.7% on ADE; combining attribute and relationship knowledge further boosts detection.The relationship module performs similarly, with a slightly higher result on VG3000.Source: §4.2 · p. 8
  • The implicit spatial module alone improves results by around 1.5% on VG1000, 0.3% on VG3000, and 0.8% on ADE, though it is less effective than attribute and relationship modules.The paper reports that unsupervised spatial knowledge still helps recognition through undefined knowledge.Source: §4.2 · p. 8
  • The VG external knowledge graph can improve ADE results, and the authors state that datasets with overlapping categories can share an existing knowledge graph.The modules can also be added to diverse detection systems.Source: §4.2 · p. 8
  • A t-SNE analysis of 8,000 VG1000 images places categories with similar attributes or spatial relationships near one another, with clearer clustering in the explicitly supervised attribute module.The modules propagate information in one pass across learned graph edges and nodes; higher-order transformations showed no significant improvement.Source: §4.2 · pp. 8–9, Figure 5 · p. 9

5 Conclusion

The authors conclude that HKRM’s two general knowledge modules can be applied to existing detection systems and that experiments indicate the approach can alleviate large-scale object-detection problems. They identify parameter reduction and further knowledge and dataset experiments as future work.

5 Conclusion

  • Experiments and analysis indicate that HKRM can alleviate problems in large-scale object detection.Source: §5 · p. 9
  • HKRM combines an externally supervised module that embeds knowledge with an implicit module that learns knowledge without explicit definitions.Source: §5 · p. 9
  • Both modules can be applied to an original detection system to improve detection performance.Source: §5 · p. 9
  • Future work proposes Cholesky re-parameterization to reduce module parameters by half and experiments with word embeddings and the Open Images Dataset.Source: §5 · p. 9

Related papers

Loading 1810.12681v1…