PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Weakly Supervised Object Localization Methods Right

Junsuk Choe, Seong Joon Oh, Seungho Lee, Sanghyuk Chun, Zeynep Akata, Hyunjung Shim

arXiv:2001.07437v2cs.CVcs.LG

TL;DR

WSOL aims to localize objects using only image-level labels, but the paper argues this setup is ill-posed and that reported progress is confounded by hidden localization supervision. It proposes controlled evaluation with a small held-out fully supervised set and finds that recent methods offer little improvement over CAM, while few-shot learning remains stronger.

  • Problem

    WSOL relies on image-level labels, yet localization methods use implicit full supervision through hyperparameter tuning and model selection, making the task ill-posed without localization supervision.

  • Method

    The paper introduces a protocol with a small held-out fully supervised set and threshold-independent evaluation metrics for fair WSOL comparison.

  • Results

    Recent WSOL methods have not made major improvements over CAM under matched validation and evaluation conditions, while few-shot learning can outperform them.

  • Takeaways & Limitations

    The findings suggest using full supervision directly for model fitting may be preferable to using it only for hyperparameter search.

  • Takeaways & Limitations

    Comparisons can differ because score-map calibration, amounts of implicit full supervision, and training settings are confounded.

Abstract

from arXiv · show

Weakly-supervised object localization (WSOL) has gained popularity over the last years for its promise to train localization models with only image-level labels. Since the seminal WSOL work of class activation mapping (CAM), the field has focused on how to expand the attention regions to cover objects more broadly and localize them better. However, these strategies rely on full localization supervision to validate hyperparameters and for model selection, which is in principle prohibited under the WSOL setup. In this paper, we argue that WSOL task is ill-posed with only image-level labels, and propose a new evaluation protocol where full supervision is limited to only a small held-out set not overlapping with the test set. We observe that, under our protocol, the five most recent WSOL methods have not made a major improvement over the CAM baseline. Moreover, we report that existing WSOL methods have not reached the few-shot learning baseline, where the full-supervision at validation time is used for model training instead. Based on our findings, we discuss some future directions for WSOL.

1. Introduction

WSOL seeks object localization from image-level labels, but reported progress beyond CAM is confounded by implicit full supervision and threshold selection. The paper argues for a held-out fully supervised protocol and reports limited gains over CAM.

  • WSOL uses image-level labels to avoid costly object-level annotations, motivating substantial research interest.
  • CAM localizes target objects using classifier activations, but its focus on discriminative object parts limits full-object coverage.
  • The paper argues WSOL is ill-posed without localization supervision and proposes a small held-out fully supervised set for learners.
  • The protocol fixes full supervision for hyperparameter search and finds no significant progress beyond CAM, while few-shot learning baselines can outperform WSOL methods.

2. Related Work

Related work frames WSOL through weakly supervised localization and multiple-instance learning, while distinguishing it from neighboring vision and explainability tasks.

  • Object localization differs from segmentation and detection by producing a mask or box for an object of a single class.
  • WSOL is commonly interpreted as multiple-instance learning over image patches, with image-level labels supervising the patch set collectively.
  • WSOL also shares similarities with visual input attribution, which analyzes pixels contributing to image classification results.

3. Problem Formulation of WSOL

The paper formulates WSOL as patch classification under multiple-instance learning and shows that image-level supervision can make localization fundamentally ill-posed. It identifies cue imbalance as the failure condition and motivates data-centric remedies and controlled supervision.

  • 3.1. WSOL Task as Multiple Instance Learning: WSOL is defined as predicting a dense binary object mask from images carrying only global image-level labels.
  • 3.1. WSOL Task as Multiple Instance Learning: An image can be represented as sliding-window patches whose local object-presence labels are collectively supervised by one image-level label.
  • 3.1. WSOL Task as Multiple Instance Learning: The task seeks a score function s(X) whose thresholded values approximate patch labels, often using the posterior s(X) = p(Y |X).
  • 3.2. When is WSOL ill-posed?: WSOL can be unsolvable even with the exact image-level posterior when background cues associate more strongly with the target than foreground cues.
  • 3.2. When is WSOL ill-posed?: The analysis represents patches by cue labels and distinguishes foreground cues M_fg from background cues M_bg.
  • 3.2. When is WSOL ill-posed?: A single threshold perfectly recovers localization only when foreground posterior likelihood is at least the background likelihood almost surely.
  • 3.2. When is WSOL ill-posed?: If a background cue predicts the image label better than a foreground cue, no WSOL method can localize correctly; duck feet versus water illustrates this case.
  • 3.2. When is WSOL ill-posed?: The paper proposes enriching positive data with underrepresented foreground cues and negative data with target-correlated backgrounds, while permitting limited localization labels for tuning.

4. Evaluation Protocol for WSOL

The protocol separates weakly supervised training, full-supervision hyperparameter selection, and held-out testing, while introducing threshold-independent metrics for localization quality.

  • Evaluation metrics: The evaluation measures localization separately from classification by using score maps for ground-truth classes rather than joint classification-localization accuracy.
  • Evaluation metrics: WSOL score maps are thresholded into binary masks and tight boxes, but fixed thresholds can disadvantage methods because optimal thresholds depend on data and architecture.
  • Evaluation metrics: PxAP summarizes pixel precision-recall performance across thresholds, allowing users to select an operating point for downstream applications.
  • Evaluation metrics: MaxBoxAcc reports the maximum box accuracy over score-map thresholds at a specified IoU threshold, with δ set to 0.5 in this paper.
  • Data splits and hyperparameter search: The protocol uses disjoint train-weaksup, train-fullsup, and test splits, reserving full supervision for search, selection, ablations, or fitting while keeping test data separate.
  • Data splits and hyperparameter search: The OpenImages benchmark contains 29,819 weakly supervised training images, 2,500 fully supervised validation images, and 5,000 test images across 100 classes.

5. Experiments

Under matched evaluation conditions, later WSOL methods show little improvement over CAM, while threshold selection and hyperparameter sensitivity substantially affect reported performance. Few-shot learning often outperforms WSOL when even limited full supervision is available.

  • Experimental setup: Six WSOL methods were evaluated across three backbones and three datasets using 30 random hyperparameter trials per method–backbone–dataset tuple.The search used the train-fullsup split and required about 9,000 GPU hours.
  • Experimental setup: Hyperparameters were optimized on an identical train-fullsup split, and test results were reported relative to vanilla CAM under a matched computational budget.The comparison used (10,5,5) fully supervised samples per class for ImageNet, CUB, and OpenImages.
  • Comparison of WSOL methods: Later WSOL methods produced no major improvement over CAM: the best overall gain was HaS at +0.2pp total mean, while only CutMix improved on ImageNet and OpenImages.ADL gained +2.0pp on CUB but did not perform well on other benchmarks.
  • Score calibration and thresholding: Fixed operating thresholds can create apparent gains without improving score maps, because methods have different score distributions and optimal thresholds.The paper therefore evaluates threshold-independent MaxBoxAcc and PxAP metrics.
  • Hyperparameter analysis: Vanilla CAM was among the least hyperparameter-sensitive methods, with σ = 1.5 on OpenImages, whereas ACoL and ADL showed greater variance, including σ = 11.9 and 9.8 on CUB.The authors suggest vanilla CAM when no full supervision is available.
  • Few-shot learning baselines: Few-shot learning surpassed vanilla CAM with 10, 5, and 5 fully labeled samples per class on ImageNet, CUB, and OpenImages, respectively.On CUB, mean FSL accuracy reached 92.0%, compared with 70.8% for maximal WSOL performance.

6. Discussion and Conclusion

The paper argues that WSOL is ill-posed without localization supervision and proposes data-centric and semi-weakly supervised directions to address this limitation. Its empirical conclusions challenge post-CAM progress and suggest using scarce full supervision for model fitting rather than hyperparameter search.

  • Discussion and Conclusion: WSOL is ill-posed without localization supervision because image-level labels do not uniquely determine object extent.The paper frames WSOL as a multiple-instance problem in which only collective image labels are observed.
  • Discussion and Conclusion: CAM remains not worse than follow-up WSOL methods under the paper’s evaluation protocol.The conclusion contrasts this finding with prior reported improvements over CAM.
  • Discussion and Conclusion: Full supervision may be more useful for model fitting than hyperparameter search in WSOL.This conclusion is presented as a qualified empirical finding.
  • Future research directions: The paper proposes semi-weakly supervised object localization for methods combining weak and full supervision.It also suggests adding more background-class images to help resolve ill-posedness.
  • Broader implications: The evaluation concerns broader settings where model selection and hyperparameter fitting implicitly provide full supervision.Examples include weakly supervised detection and segmentation, zero-shot learning, and unsupervised disentanglement.
  • Foreground-background posterior ratio: Increasing foreground cues and confusing background cues can improve the foreground-background posterior ratio.The duck example contrasts feet as foreground cues with water as a correlated background cue.

A.2. Proof for the ill-posedness lemma

The appendix defines pixel-wise localization accuracy and characterizes when perfect thresholded localization is possible. It also documents per-image score normalization and optimal-threshold box evaluation.

  • Proof for the ill-posedness lemma: Pixel-wise localization accuracy combines foreground recall and background specificity weighted by their pixel probabilities.The definition evaluates correct pixel-wise predictions under threshold τ.
  • Proof for the ill-posedness lemma: Perfect pixel-wise accuracy exists exactly when the foreground-background posterior ratio is at least 1 almost surely.The condition is stated conditionally on observing foreground and background patches.
  • Score calibration and thresholding: WSOL score maps are normalized per image before thresholding, using either max or min-max normalization.The appendix states that this compensates for large differences in score ranges across images.

B.2. Data preparation

The paper constructs held-out, fully annotated data for CUB, ImageNet, and OpenImages to support WSOL hyperparameter search and evaluation. OpenImages is substantially processed to address multilabeling, class imbalance, and incomplete instance masks.

  • Data contributions: The data contributions include five annotated CUB images per class, annotated ImageNetV2 images, and a processed OpenImages WSOL split.The resources cover training, hyperparameter search, and evaluation with localization annotations.
  • ImageNet: ImageNetV2 provides three 10,000-image subsets used as new test sets for ImageNet-1k trained models.The subsets are MatchedFrequency, Threshold0.7, and TopImages.
  • CUB: CUB held-out images are collected from Flickr, deduplicated, manually pruned, classified, and given tight bird bounding boxes.The collection targets five images for each of 200 fine-grained bird classes.
  • OpenImages: OpenImages is unsuitable in its original form because it is multilabeled, imbalanced, and contains many missing instance masks.These properties differ substantially from CUB and ImageNet for WSOL benchmarking.
  • OpenImages: The processed OpenImages split rejects 34.5% of samples and retains 100 classes meeting minimum train, validation, and test counts.It also constructs binary foreground masks from unions of relevant instance masks and introduces ignore regions.
  • Hyperparameter search: Hyperparameter rankings are largely preserved between full and reduced training settings, with Kendall’s tau equal to 0.743.The reduced ImageNet training set uses 10% of train-weaksup for computational efficiency.

C.1. Prior WSOL methods and hyperparameters

The appendix summarizes six WSOL methods and their hyperparameters, spanning CAM, erasing-based architectures, self-guidance, and data augmentation. Their search spaces include learning, resolution, dropout, erasing, guidance, and mixing parameters.

  • CAM: CAM uses a fully convolutional classifier with global average pooling and scores localization from pre-pooling logits.Its hyperparameters include learning rate and score-map resolution.
  • Data augmentation methods: HaS randomly drops grid-like image patches, with drop rate and drop area controlling erasure frequency and patch size.The drop-area range spans from no grid to treating the full image as one patch.
  • Architectural methods: ACoL adds a second classification head that learns from feature maps after high-score regions identified by CAM are erased.Its erasing threshold controls how much is removed from the internal feature map.
  • Hyperparameter evaluation: The appendix reports hyperparameter-ranking comparisons between train-fullsup and test using Kendall’s tau.Table 4 is explicitly described as measuring ranking preservation across these splits.
  • Architectural methods: SPG uses three auxiliary branches and low- and high-score thresholds to separate foreground, background, and ignored regions.Pixels between the two thresholds are ignored during guidance.
  • Architectural methods: ADL alternates between an attention-derived drop mask and an importance map during training.The drop rate and erasing threshold determine the dropout behavior.
  • Data augmentation methods: CutMix cuts and pastes image patches while mixing target labels, controlled by size-prior α and mix rate r.The size prior is sampled through a beta distribution parameterized by α.

C.2. Classification results of WSOL methods

Classification and localization performance can diverge in WSOL, making classification-entangled evaluation potentially misleading. The reported results and table setup therefore emphasize localization-only metrics for comparison and model selection.

  • Top-1 localization accuracy combines classification and localization, so increases may reflect improved classification rather than better localization.
  • 26.8% to 74.5% classification accuracy on CUB shows substantial variation when hyperparameters are selected for localization.The selection procedure uses train-fullsup localization performance, and localization can peak before classifiers are sufficiently trained.
  • Localization-only metrics such as MaxBoxAcc and PxAP are recommended for WSOL model selection and evaluation.
  • The reproduced results compare top-1 localization accuracy with GT-known localization metrics across ImageNet, CUB, and OpenImages using VGG, InceptionV3, and ResNet50.The table includes six WSOL methods and separates the reported metric families.

C.3. Reproducing prior WSOL results

Reproduction analysis shows that training and threshold choices materially affect WSOL comparisons. The authors standardize training conditions and use threshold-aware evaluation to expose differences in reproducibility and score calibration.

  • Approximately 10 percentage points in MaxBoxAcc can depend on training epochs and batch size.Because prior papers do not consistently report training details, the authors argue that methods should share a common training budget.
  • The reimplementations exceed a prior CAM ImageNet Inception result from 62.7 to 65.3 under GT-known metrics.The authors attribute this partly to MaxBoxAcc and PxAP using the best operating thresholds.
  • ADL on CUB with VGG drops from 52.4 to 39.2 in reproduced top-1 localization accuracy.The authors attribute the difference to fewer training epochs and decreased classification accuracy.
  • Optimal operating thresholds vary substantially across datasets and architectures, while MaxBoxAcc is relatively stable across methods on ImageNet.OpenImages PxPrec-PxRec curves also show little difference among methods.
  • Different WSOL methods produce qualitatively different score maps, but visual inspection does not reliably predict optimal IoU.The authors therefore emphasize objective threshold-setting procedures.
  • Hyperparameter choices change performance; CAM is comparatively stable, whereas ACoL and ADL are more sensitive.On CUB, random hyperparameter selection can underperform the center-Gaussian baseline.
  • Performance generally increases during training, but some methods reach their best localization at early iterations.The paper notes this can occur for CUB SPG and OpenImages CAM, where continued classification training may hurt localization.

C.7. Evaluating WSOL methods with MaxBoxAccV2

Re-evaluation with MaxBoxAccV2 preserves the paper’s central conclusions: WSOL methods show no significant progress beyond CAM, and few-shot supervised baselines perform better with equal fully supervised samples.

  • No significant WSOL performance progress beyond vanilla CAM is observed with MaxBoxAccV2.The re-evaluation covers six recently proposed methods under the same training configurations as the main evaluation.
  • Few-shot learning baselines outperform existing WSOL methods when given the same amount of fully supervised samples.
  • The supplementary evaluation includes threshold-performance plots, score maps, hyperparameter trials, and learning curves across three datasets and architectures.These materials extend the main-paper analyses for calibration, visualization, hyperparameter sensitivity, and training dynamics.
Loading 2001.07437v2…