PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Judge an Object by Its Context: Learning to Overcome Contextual Bias

Krishna Kumar Singh, Dhruv Mahajan, Kristen Grauman, Yong Jae Lee, Matt Feiszli, Deepti Ghadiyaram

arXiv:2001.03152v2cs.CVcs.LG

TL;DR

Contextual co-occurrence bias can make classifiers fail when a category appears without its usual context. The paper uses CAM-based localization and feature splitting to decorrelate category representations from context, improving hard-case recognition across object and attribute classification while preserving overall performance.

  • Problem

    Context-biased classifiers may fail to recognize categories away from their typical context, such as “ski” without “person,” while needing to retain performance when context is present.

  • Method

    The paper combines CAM-based spatial-overlap minimization with a dedicated feature subspace learned from context-absent instances alongside a joint category-context feature space.

  • Results

    Across object and attribute classification on 4 datasets, the methods improve recognition in hard context-absent cases; feature splitting gains 4.3% on the exclusive test set and drops only 0.2% on the co-occurring split.

  • Takeaways & Limitations

    Decorrelating category features from context reduces contextual bias while maintaining overall performance and generalizing to unseen datasets.

  • Takeaways & Limitations

    The CAM-based approach can hurt co-occurring performance because discouraging spatial overlap may discard complementary information from context regions, and it is unsuitable for some occluding category pairs.

Abstract

from arXiv · show

Existing models often leverage co-occurrences between objects and their context to improve recognition accuracy. However, strongly relying on context risks a model's generalizability, especially when typical co-occurrence patterns are absent. This work focuses on addressing such contextual biases to improve the robustness of the learnt feature representations. Our goal is to accurately recognize a category in the absence of its context, without compromising on performance when it co-occurs with context. Our key idea is to decorrelate feature representations of a category from its co-occurring context. We achieve this by learning a feature subspace that explicitly represents categories occurring in the absence of context along side a joint feature subspace that represents both categories and context. Our very simple yet effective method is extensible to two multi-label tasks -- object and attribute classification. On 4 challenging datasets, we demonstrate the effectiveness of our method in reducing contextual bias.

1. Introduction

Context is a useful visual cue, but biased co-occurrences can make classifiers depend on context and fail when it is absent. The paper proposes CAM-based separation and feature splitting to improve recognition without sacrificing co-occurring cases.

  • Motivation: Context aids recognition, but dataset co-occurrences can cause models to rely on contextual objects rather than the target category.Examples include microwaves with kitchen objects and attributes such as “trapeze” with “striped.”
  • Motivation: Contextual bias can cause failures both when the target appears in a different context and when the context object is incorrectly hallucinated.The introduction illustrates these risks for “microwave” and “refrigerator.”
  • Goal: The paper aims to recognize categories both with and without context while preserving performance when the category and context co-occur.The goal applies to strongly co-occurring category pairs and extends to object and attribute classification.
  • Methods: The first method uses class activation maps as weak location annotations and minimizes spatial overlap between co-occurring categories.This encourages the network to rely more on a category’s pixels than on its context.
  • Methods: The second method decorrelates category and context representations by adding a context-absent feature subspace alongside a joint feature space.The dedicated subspace is learned from instances where the biased category occurs without its usual context, while the full space remains available otherwise.
  • Results: Across two tasks and four datasets, both methods improve hard cases where categories occur away from typical context, while the framework is designed not to harm co-occurring recognition.The paper reports significant boosts over strong baselines for context-absent cases.

2. Related work

Prior work addresses dataset and representation bias through resampling, adaptation, adversarial learning, and task-specific methods, but contextual bias in visual recognition remains underexplored. Existing alternatives also require information or annotations unavailable to this image-based setting.

  • Addressing biases: Prior bias-mitigation work includes domain adaptation, data resampling, and adversarial learning, but contextual bias remains relatively underexplored.Adversarial methods cannot simply remove context because context may remain useful for recognition.
  • Contextual bias methods: Related approaches target video action recognition, attribute correlations, or object detection, but rely on temporal information, semantic groupings, or bounding-box annotations.The proposed image-recognition setting does not require such additional information.
  • Contextual bias measurement: Figure 2 quantifies contextual bias through the high co-occurrence of a category b with a context category c.The figure is presented as a measurement of bias rather than as an evaluation comparison.
  • Alternative formulations: Rare context-absent images can resemble few-shot or class-imbalance settings, but treating them as a separate class or simply upweighting them can be sub-optimal.The paper distinguishes these strategies from its use of contextual relationships.

3. Problem setup

The problem setup assumes that a few category pairs are strongly co-occurrence-biased in training data, even though test data are not assumed to share that distribution. The method identifies these pairs using context dependence and co-occurrence frequency before training with weak labels.

  • Problem setup: The framework targets categories whose training distributions are biased by frequent co-occurrence with particular context categories, without assuming the same bias in test data.For example, COCO-Stuff contains 2209 “ski”-with-“person” images but only 29 “ski”-without-“person” images.
  • Problem setup: The goal is to recognize a biased category in isolation while preserving performance when it co-occurs with its usual context.This is defined for an unbiased test distribution.
  • Identifying biased categories: The method constructs a set S of K biased category pairs (b_j, c_j), where b is the biased category and c is its context.Only a few of the M vocabulary categories are assumed to suffer from this bias.
  • Identifying biased categories: A context category is considered strongly biasing when the prediction probability of b drops substantially without c and b frequently co-occurs with c.The bias measure compares average predictions for b with and without a candidate context category.
  • Method inputs: Both proposed methods take weak multi-label training images and the selected biased-pair set as input, without spatial annotations.The input labels specify which categories are present but not their image locations.

4. Approach

The approach reduces contextual bias through CAM-based spatial decorrelation and feature splitting that selectively suppresses context for categories occurring without their typical context.

  • 4.1. CAM as “weak” location annotation: CAMs provide weak location annotations from class labels, enabling the method to minimize spatial overlap between biased categories and co-occurring context without ground-truth locations.The CAM objective is regularized toward offline CAMs to avoid drifting from actual category regions.
  • 4.1. CAM as “weak” location annotation: The CAM-based method improves recognition when biased categories occur without context, but can hurt co-occurring cases because useful contextual information is excluded.This limitation motivates separating category and context in feature space rather than constraining their spatial overlap.
  • 4.2. Feature splitting and selective context suppression: Feature splitting divides representations into x_o and x_s, suppressing x_s and its gradients for context-absent examples while using both subspaces when context co-occurs.At inference, the entire feature space is used; the method selectively suppresses context only during training.
  • 4.2. Feature splitting and selective context suppression: The feature-splitting method uses weighted binary cross-entropy because context-absent examples are rare, weighting them according to the ratio of context-present to context-absent training instances.The context subspace is set to a recent mini-batch average for more stable training.
  • 4.3. Training setup: The methods target selected biased category pairs, identified from frequent co-occurrences and a train-validation bias estimate, and are evaluated across object and attribute classification datasets.The setup uses K = 20 biased pairs and standard multi-label classification loss.

5. Experiments

Across object and attribute classification, the experiments show that the proposed methods improve recognition when biased categories occur without typical context while generally preserving co-occurring performance. Qualitative analyses and cross-dataset evaluation further indicate that feature-split representations separate category-specific and contextual information and generalize to unusual contexts.

  • 5.1.1 Overall Results: Class balancing loss yields only marginal gains, while weighting exclusive cases improves exclusive performance but substantially hurts co-occurring performance.Weighted loss reaches 30.4% versus ours-feature-split's 28.8% on the exclusive split, but falls to 60.8% versus 66.0% on the co-occurring split.
  • 5.1.1 Overall Results: 4.3%: ours-feature-split outperforms standard on the exclusive COCO-Stuff test split while dropping only 0.2% on the co-occurring split.Both methods outperform standard on the exclusive split, by 1.9% for ours-CAM and 4.3% for ours-feature-split; ours-CAM has a larger co-occurring-performance drop.
  • 5.1.2 Qualitative Analysis: Visualizations show ours-CAM focuses on object pixels, whereas ours-feature-split additionally uses context and succeeds on biased objects away from typical context.Ours-feature-split recognizes examples such as skateboard and snowboard without person, and microwave without oven; both methods also have failures in difficult low-light or small-object cases.
  • 5.1.2 Qualitative Analysis: The feature-split activation maps separate category and context: Wo focuses on the category, while Ws captures co-occurring context.Examples include car or handbag for category-specific features and road or person for contextual features.
  • 5.2. Cross dataset experiment on UnRel: Without fine-tuning on UnRel, both methods outperform standard by large margins, indicating generalization to objects in unusual contexts.The cross-dataset test uses only three UnRel categories overlapping with the biased COCO-Stuff categories.
  • 5.3. Attribute Classification: Ours-feature-split improves attribute classification on DeepFashion and Animals with Attributes, including 21.2% and 17.4% top-3 recall gains for trapeze and bell.The method improves both exclusive and co-occurring splits, while the CAM-based approach is not applicable to attributes because they lack well-defined spatial extents.

6. Conclusion

The paper shows that contextual bias harms recognition outside typical contexts and proposes two methods that improve exclusive-case recognition while preserving overall performance.

  • Both methods improve recognition when biased classes occur away from their co-occurring context while maintaining overall performance.They also generalize to new unseen datasets and outperform standard methods.
  • The framework currently addresses contextual bias between pairs of categories.Future work could incorporate more scene or category information and extend the methods to object detection and video action recognition.
  • Standard classifiers perform poorly when biased categories occur away from their typical context.

7. Additional implementation Details

The additional implementation details specify how biased category pairs are selected, how bias is measured across datasets, and how αmin is chosen for inference.

  • Biased pairs are selected when prediction probability drops without the context and the context co-occurs frequently with the biased category.The co-occurring class appears at least 20% of the time for COCO-Stuff and Animals with Attributes, and 10% for DeepFashion.
  • Bias is measured using dataset-specific splits: an 80–20 split for COCO-Stuff, validation data for DeepFashion, and test data for Animals with Attributes.Animals with Attributes uses the test set because its distribution differs from training data.
  • αmin is set to 3 for COCO-Stuff and Animals with Attributes and 5 for DeepFashion through cross-validation.Inference takes 0.2 ms per image in a single forward pass on one Titan X GPU.

8. More results

Across object and attribute datasets, the proposed methods improve recognition when biased categories occur without context while preserving performance in co-occurring cases and reducing feature dependence on context.

  • Baseline analysis: The split biased baseline cannot fully exploit co-occurring and exclusive images simultaneously, explaining its inferior performance relative to both proposed methods.
  • Comparison with split biased: 28.8 vs. 19.1 COCO-Stuff mAP: ours-feature-split outperforms split biased, while ours-CAM reaches 26.4 vs. 19.1.The comparison is reported for the COCO-Stuff setting against the split biased baseline.
  • Performance on non-biased classes: 0.02% lower mAP on non-biased object and stuff classes: the approach incurs only a marginal performance reduction while improving biased-category recognition outside context.
  • Feature separation: Lower cosine similarity than standard indicates that both methods learn more distinct representations for biased category pairs.The weight similarity measure is used to assess whether the category and context capture distinct information.
  • Per-class results: ours-feature-split improves exclusive-case performance while maintaining co-occurring performance across per-class results for COCO-Stuff, DeepFashion, and Animals with Attributes.
  • Ablation study: Higher fractions of exclusive training images benefit all approaches, but the proposed methods consistently outperform standard, especially under skewed training distributions.
  • DeepFashion: DeepFashion top-3 recall is 9.2 vs. 6.0 for ours-feature-split and remove co-occur labels on exclusive attributes, while co-occurring recall is 20.1 vs. 20.4.ours-feature-split outperforms all baselines on the exclusive test set without materially sacrificing co-occurring performance.
  • Animals with Attributes: On Animals with Attributes, ours-feature-split improves exclusive cases without the severe co-occurring-case harm caused by remove co-occur images.
Loading 2001.03152v2…