PaperlayerView exact revision on arXiv

Source-linked AI summary

Min-Entropy Latent Model for Weakly Supervised Object Detection

Fang Wan, Pengxu Wei, Zhenjun Han, Jianbin Jiao, Qixiang Ye

arXiv:1902.06057v1cs.CV

TL;DR

Weakly supervised object detection must learn object locations and detectors from image-level supervision, whose mismatch with the object-level objective creates localization randomness and detector ambiguity. MELM combines clique partitioning, global and local min-entropy models, and recurrent learning with continuation optimization. The paper reports improved weakly supervised detection, localization, and image classification performance, while noting difficulty precisely localizing classes with large appearance variance, especially person.

  • Problem

    Weakly supervised object detection lacks object-level annotations while learning object locations and detectors, producing localization randomness and detector ambiguity.

  • Method

    MELM partitions redundant proposals into cliques, applies global and local min-entropy models, and uses recurrent learning with continuation optimization.

  • Results

    MELM improved weakly supervised detection, localization, and image classification performance against state-of-the-art approaches.

  • Takeaways & Limitations

    Min-entropy reduces the variance of positive instances and alleviates detector ambiguity, while recurrent learning provides insights for weakly supervised learning.

  • Takeaways & Limitations

    MELM failed on the person class and often precisely localized classes with large appearance variance below IoU 0.5.

Abstract

from arXiv · show

Weakly supervised object detection is a challenging task when provided with image category supervision but required to learn, at the same time, object locations and object detectors. The inconsistency between the weak supervision and learning objectives introduces significant randomness to object locations and ambiguity to detectors. In this paper, a min-entropy latent model (MELM) is proposed for weakly supervised object detection. Min-entropy serves as a model to learn object locations and a metric to measure the randomness of object localization during learning. It aims to principally reduce the variance of learned instances and alleviate the ambiguity of detectors. MELM is decomposed into three components including proposal clique partition, object clique discovery, and object localization. MELM is optimized with a recurrent learning algorithm, which leverages continuation optimization to solve the challenging non-convexity problem. Experiments demonstrate that MELM significantly improves the performance of weakly supervised object detection, weakly supervised object localization, and image classification, against the state-of-the-art approaches.

1 INTRODUCTION

Weakly supervised object detection uses only image-level labels while learning object locations and detectors, creating localization randomness and detector ambiguity. MELM addresses these issues with clique-based min-entropy modeling and recurrent learning.

  • Weakly supervised object detection learns detectors from image-level annotations without object-level bounding boxes.This avoids the laborious and expensive collection of bounding-box annotations.
  • Existing approaches select instances from redundant proposals while learning classifiers, but their non-convex objectives can produce unstable object locations and ambiguous detectors.Object parts may switch during learning because different regions can minimize image classification loss.
  • MELM collects instances with minimum randomness using proposal cliques, global min-entropy object clique discovery, and local min-entropy object localization.Cliques group spatially and class-related proposals, while the model suppresses object parts and activates object extent.
  • MELM uses recurrent learning to transfer image-category supervision to object locations through repeated interactions between classification and detection.The recurrent procedure can discover multiple objects from a single image when they exist.
  • The paper reports state-of-the-art performance for weakly supervised detection, localization, and image classification.The contribution list identifies performance across all three tasks as a principal result.

2 RELATED WORK

Earlier WSOD work used clustering, latent-variable learning, and MIL to select proposals and estimate classifiers, but localization remained vulnerable to noise, random solutions, and annotation-objective mismatch. MELM combines proposal cliques, min-entropy models, and recurrent feedback to address these limitations.

  • Pipelined WSOD methods decompose images into proposals before applying clustering, latent-variable learning, or multiple instance learning.Deep learning later integrated these strategies into MIL networks.
  • Clustering: Clustering methods can identify proposal sub-categories but may incorporate substantial clutter noise into a true positive cluster.The limitation follows from assuming that object instances form compact clusters.
  • Latent Variable Learning: Latent-variable methods address non-convex learning with regularization, yet can become trapped in poor local minima that falsely localize parts or backgrounds.Prior work introduced symmetry, mutual exclusion, smoothing, and convex clustering to seek stronger minima.
  • Multiple Instance Learning: MIL treats each image as a bag and iteratively selects high-scored instances, but remains susceptible to random poor solutions.Multi-fold MIL reduces this randomness through dataset division and cross-validation.
  • Multiple Instance Learning: MIL networks learn effective image representations but have limited localization because image-level supervision also activates object parts and backgrounds.MELM instead introduces proposal cliques and min-entropy latent models to reduce localization randomness.
  • Multiple Instance Learning: MELM inherits multi-branch MIL architectures and adds recurrent learning so network branches can benefit from object-score feedback.This feedback helps translate image labels into object locations.

3.1 Overview

MELM targets localization randomness caused by proposal redundancy and uncertain object locations in non-convex WSOD learning. It partitions proposals into cliques, discovers discriminative cliques with global min-entropy, and localizes objects with a local min-entropy model.

  • WSOD localization randomness arises from proposal redundancy, many local minima, and switching among uncertain object parts.The supervision is image-level while the learning objective concerns object-level classifiers.
  • MELM partitions redundant proposals into cliques of spatially overlapping and class-related regions.This reduces proposal redundancy before min-entropy optimization.
  • A global min-entropy model discovers discriminative object cliques and suppresses cliques lacking discriminative information.The discovered cliques are used to activate true object extent.
  • A local min-entropy model estimates pseudo-objects and spatial-neighbor hard negatives within discovered cliques.These estimates are used to improve object localization during learning.

3.2 Min-Entropy Latent Model

MELM models weakly supervised localization with min-entropy constraints and decomposes learning into clique partition, object clique discovery, and object localization. Global and local min-entropy components progressively select object candidates, suppress parts, and improve detector localization.

  • 3.2.1 Clique partition: MELM uses high-scored proposals to form spatially overlapping, class-related cliques that collect object regions and parts.The partition operates on selected proposals and targets activation of the true object extent.
  • 3.2.2 Object clique discovery with global min-entropy: The global min-entropy model discovers discriminative object cliques while encouraging their evolution with minimum localization randomness.Its classification-related weight links clique selection to proposal object scores and image-level discrimination.
  • 3.2.3 Object localization with local min-entropy: The discovered cliques initialize localization but may contain object parts and partial objects because clique discovery does not precisely localize objects.This limitation follows from optimizing discrimination between positive and negative images rather than precise localization.
  • 3.2.3 Object localization with local min-entropy: The local min-entropy model estimates pseudo-objects and hard negatives from each clique’s spatial neighborhood.Proposals close to the selected location are treated as likely objects, while proposals with IoU below 0.5 are hard negatives.
  • 3.2.3 Object localization with local min-entropy: Optimizing the localization branch produces sparse high-probability proposals, suppresses object parts, and progressively improves detector localization.The local model uses soft proposal labeling based on spatial overlap.

3.3 Model Implementation

MELM is implemented as an integrated deep network with clique partitioning and separate branches for object clique discovery and localization. Recurrent forward and backward updates refine proposal scores and detector parameters through repeated learning iterations.

  • 3.4 Model implementation: MELM adds a clique partition module and two branches on top of fully connected network layers.The discovery branch optimizes global entropy and classification, while the localization branch optimizes local entropy and pseudo-object detection loss.
  • 3.4 Model implementation: Each recurrent iteration extracts proposal features, partitions cliques, discovers object cliques, localizes objects, and updates network parameters.The algorithm then recomputes proposal object scores using the updated parameters.
  • 3.4 Model implementation: ROI pooling over CONV5 provides efficient proposal features during learning.Forward propagation selects sparse object instances, and back-propagation optimizes the network parameters.
  • 3.4 Model implementation: The object probability of each proposal is recurrently aggregated by multiplying it with the probability learned in the preceding iteration.During detection, learned soft-max and fully connected layers classify proposals and localize objects.

3.4 Model Learning

MELM transfers image-level supervision to object locations through an integrated recurrent learning procedure. Its forward pass discovers and localizes pseudo-objects for detector learning, while accumulated branches support multiple objects and appearance diversity.

  • 3.4 Model learning: The learning objective transfers image category supervision to object locations under minimum-localization-randomness constraints.The constraints express the min-entropy objective within the weakly supervised setting.
  • 3.4 Model learning: Forward propagation discovers object cliques and localizes objects for pseudo-object detector learning, while back-propagation updates the network.The learned detectors assign new object probabilities that aggregate proposal scores.
  • 3.4 Model learning: Accumulated recurrent learning adds localization branches that accumulate objects across iterations.This supports localizing multiple objects in one image and learning diverse object appearances.

3.5 Model analysis

MELM uses continuation optimization and recurrent prediction-correction to address the non-convexity of weakly supervised learning. Its approximate clique-level objective reduces the solution space before object localization and iteratively approaches the original objective.

  • MELM uses continuation optimization to alleviate the non-convexity problem.
  • The model replaces direct optimization of a complex objective with a smoother approximate function before approaching the original objective.The approximate function is controlled by λ, which ranges from 0 to 1 and controls smoothness.
  • The recurrent procedure iteratively defines predictions and corrections to approximate the original objective and approach the globally optimal solution θ∗.
  • Clique partition reduces each image’s solution space from thousands of proposals to tens of cliques by averaging proposal class probabilities.
  • Recurrent learning decomposes weakly supervised learning into object clique discovery as prediction and object localization as correction.The correction model fills the gap between clique discovery and object localization.

3.6 Object Detection

MELM detects objects by extracting proposal features and assigning object scores during object localization. The procedure operates on redundant proposals generated by Selective Search or EdgeBox.

  • MELM detects objects by extracting proposal features with ROI pooling and assigning object scores during localization.Test images use redundant proposals from Selective Search or EdgeBox before feature extraction.

4 EXPERIMENTS

Experiments show that MELM reduces localization randomness while improving detection, localization, and classification performance across standard and large-scale datasets. Ablations indicate that min-entropy components, recurrent learning, and accumulated branches each contribute to performance, though localization remains difficult for high-variance classes.

  • Randomness Analysis: MELM’s cliques collected object extents early, suppressed object parts through localization, and detected objects accurately after recurrent learning.Object extent was collected at Epoch 2, object parts were suppressed at Epoch 4, and accurate detection was reported at Epoch 20.
  • Randomness Analysis: MELM reduced localization randomness and achieved higher localization accuracy than WSDDN during learning, while WSDDN switched among object parts.For bicycle images, MELM eventually localized objects accurately whereas WSDDN failed to localize the true objects.
  • Ablation Experiments: Recurrent learning improved MELM-L mAP from 40.1% to 42.6%, while accumulated recurrent learning increased it further to 47.3% with three branches.Using four branches did not significantly improve performance.
  • PASCAL VOC datasets: MELM achieved 47.3% mAP on PASCAL VOC 2007 with VGG16, outperforming OICR by 6.1%, Self-Taught by 5.6%, WCCN by 4.5%, WeakRPN by 3.0%, and TS2C by 2.0%.A Fast-RCNN detector retrained with learned pseudo objects and ResNet-101 achieved 49.0% mAP.
  • PASCAL VOC datasets: MELM improved VOC 2010 and VOC 2012 detection results over several state-of-the-art methods, including 42.4% versus 37.9% against WCCN on VOC 2012 with VGG16.The largest reported class-specific gains included dining-table (+14.7%), dog (+9.6%), and cow (+8.5%).
  • Large-scale datasets: MELM improved mean CorLoc, large-scale detection, and image classification, reaching 61.4% mean CorLoc with VGG16, 13.4% versus 9.8% on ILSVRC2013, and 93.1% classification mAP with VGG16.On MS COCO, MELM outperformed SPN by 23.1% for image classification and 9.8% for pointing localization.

5 CONCLUSION

MELM integrates clique-based min-entropy modeling into deep learning for weakly supervised object detection. It reduces instance variance and detector ambiguity while improving performance across detection, localization, and image classification.

  • MELM integrates clique partition, object clique discovery, and object localization into a unified deep learning framework.These components learn latent object regions from redundant object proposals.
  • Min-entropy principally reduces the variance of positive instances and alleviates detector ambiguity.
  • The recurrent learning algorithm improves weakly supervised detection, weakly supervised localization, and image classification.It leverages continuation optimization to address the non-convexity of weakly supervised learning problems.

APPENDIX

The appendix derives gradients for the object clique discovery and object localization components. It specifies how entropy-model quantities and pseudo-label-based localization terms enter optimization.

  • The appendix introduces shorthand notation for four energy and loss functions with different inputs.
  • Object clique discovery: Object clique discovery computes gradients by differentiating the entropy model with respect to proposal probabilities and object scores.The derivation includes the partial derivatives of E(Hc, θ) with respect to p(y, h; θ) and of p(y, h; θ) with respect to s(y, h; θ).
  • Object clique discovery: The score derivative distinguishes proposals belonging to the same clique and class from those with different clique or class assignments.
  • Object localization: Object localization treats whp(y, h; θ) as a non-backpropagated pseudo-label when deriving the localization gradient.The derivative of L(h, θ) with respect to s(y, h; θ) is computed using the appendix equations.
Loading 1902.06057v1…