PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Predict Visual Attributes in the Wild

Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, Abhinav Shrivastava

arXiv:2106.09707v1cs.CV

TL;DR

Visual attribute prediction lacks a large, in-the-wild benchmark that handles many labels, sparsity, imbalance, and occlusion. The paper introduces VAW and SCoNE, combining attention-based modeling with label-expansion and contrastive techniques, and reports substantial gains over prior methods.

  • Problem

    Existing visual attribute prediction work is mostly domain-specific, while large-scale in-the-wild prediction faces many attributes, label sparsity, imbalance, and object occlusion.

  • Method

    The paper introduces VAW and SCoNE, using low- and high-level features with multi-hop attention, negative-label expansion, reweighting or resampling, and supervised attribute-aware contrastive learning.

  • Results

    SCoNE surpasses state-of-the-art models by 3.5 mAP and 5.7 overall F1 points.

  • Takeaways & Limitations

    VAW provides an in-the-wild benchmark for long-tailed multi-label prediction with limited labels, data imbalance, out-of-distribution testing, and bias-related issues.

Abstract

from arXiv · show

Visual attributes constitute a large portion of information contained in a scene. Objects can be described using a wide variety of attributes which portray their visual appearance (color, texture), geometry (shape, size, posture), and other intrinsic properties (state, action). Existing work is mostly limited to study of attribute prediction in specific domains. In this paper, we introduce a large-scale in-the-wild visual attribute prediction dataset consisting of over 927K attribute annotations for over 260K object instances. Formally, object attribute prediction is a multi-label classification problem where all attributes that apply to an object must be predicted. Our dataset poses significant challenges to existing methods due to large number of attributes, label sparsity, data imbalance, and object occlusion. To this end, we propose several techniques that systematically tackle these challenges, including a base model that utilizes both low- and high-level CNN features with multi-hop attention, reweighting and resampling techniques, a novel negative label expansion scheme, and a novel supervised attribute-aware contrastive learning algorithm. Using these techniques, we achieve near 3.7 mAP and 5.7 overall F1 points improvement over the current state of the art. Further details about the VAW dataset can be found at http://vawdataset.com/.

1. Introduction

The paper introduces VAW, a large-scale in-the-wild dataset for multi-label visual attribute prediction, and proposes SCoNE to address sparse, imbalanced, and partially labeled annotations.

  • 1. Introduction: Existing attribute-prediction datasets are limited by domain specificity, few attribute-object pairs, label noise, ambiguity, sparsity, and missing negative labels.These limitations make large-scale in-the-wild attribute prediction difficult to train and evaluate.
  • 1. Introduction: VAW contains over 927K explicitly labeled positive and negative annotations for over 260K object instances across 620 attributes.The dataset includes 2,260 object phrases and frames prediction as long-tailed, partially labeled, multi-label classification.
  • 1. Introduction: The baseline combines low- and high-level CNN features, multi-hop attention, and available segmentation masks to model heterogeneous attribute cues.The design targets differences between attributes such as color and action while localizing relevant object regions.
  • 1. Introduction: SCoNE surpasses state-of-the-art models by 3.5 mAP and 5.7 overall F1 points.SCoNE combines negative-label expansion with supervised contrastive learning and other attribute-learning techniques.
  • 1. Introduction: The paper presents VAW and SCoNE as contributions addressing difficult large-scale visual attribute prediction challenges.The stated contributions include dataset construction, a strong attention-based baseline, and techniques evaluated through extensive experiments.

2. Related Work

Related work spans attribute learning, multi-label prediction with missing labels, imbalance handling, attention, and contrastive learning, but does not fully address VAW’s combined setting.

  • Related Work: Prior general attribute datasets are limited in scope or have sparse, noisy annotations without negative labels, motivating VAW.COCO Attributes covers 29 object categories, while Visual Genome attributes are not designed as a standalone benchmark.
  • Related Work: Missing-label methods often treat unobserved labels as negative, which is unsuitable because visual attributes are generally not mutually exclusive.Alternative expert-model approaches are described as infeasible for a large-scale problem like VAW.
  • Related Work: The paper combines cost-sensitive learning and resampling to address label imbalance in VAW.These are presented as the two main approaches used in related imbalanced-data work.
  • Related Work: Segmentation masks guide attention away from irrelevant regions while additional attention maps explore surrounding object context.This attention design is enabled by the prevalence of segmentation annotations in VAW.
  • Related Work: The paper extends supervised contrastive learning to multi-label prediction so samples can be grouped by shared attributes.It characterizes this as the first attempt to apply contrastive loss to multi-label learning.

3. Visual Attributes in the Wild (VAW) Dataset

VAW is constructed from refined existing annotations and expanded with balanced attribute-object coverage, producing a dense in-the-wild benchmark with explicit negatives and segmentation masks.

  • 3. Visual Attributes in the Wild (VAW) Dataset: The dataset begins with VGPhraseCut and GQA, which provide refined attributes, segmentation masks, and cleaner scene-graph annotations.Both source datasets leverage and refine annotations from Visual Genome.
  • 3. Visual Attributes in the Wild (VAW) Dataset: The construction process selects 620 frequently occurring, manually cleaned attributes and expands instances from GQA.The goal is to predict as many commonly used object attributes as possible in the wild.
  • 3. Visual Attributes in the Wild (VAW) Dataset: For every feasible attribute-object pair, annotation collection ensures at least 50 positive or negative examples.A feasible pair has at least one positive example; already sufficiently annotated pairs are skipped to control cost.
  • 3. Visual Attributes in the Wild (VAW) Dataset: VAW contains 620 attributes describing 260,895 instances from 72,274 images, including 392,978 positive and 534,701 negative labels.The attributes cover color, material, shape, size, texture, and action; 92% of instances have segmentation masks.
  • 3. Visual Attributes in the Wild (VAW) Dataset: VAW fills a benchmark gap with denser annotations, explicit negative labels, segmentation masks, and many attribute and object categories.The comparison is made against existing in-the-wild and domain-specific visual attribute datasets.

4. Methodology

The method combines object-conditioned visual features, object localization, multi-attention aggregation, imbalance handling, negative-label expansion, and supervised attribute-aware contrastive learning for multi-label attribute prediction.

  • Image-object feature composition: Object-conditioned gating filters image features to retain attributes relevant to the object and suppress incompatible attribute-object pairs.The gate is implemented as a two-layer MLP over the object embedding and is broadcast across the feature map.
  • Object localization and multi-attention module: The baseline localizes the relevant object region from the composed feature map and aggregates multiple attention maps for attributes requiring different object regions.Localization is supervised by available segmentation masks but penalizes attention outside the object more strongly than incomplete coverage inside it.
  • Strong baseline model: The model combines low-level, image-object, and attention-pooled features before multi-label classification with explicit positive and negative labels.The architecture is designed to accommodate heterogeneous attribute cues such as color, material, texture, shape, and action.
  • Loss function and training paradigm: Reweighted BCE and repeat-factor sampling jointly address imbalance, while missing labels can be treated as low-weight soft negatives during BCE training.The reweighting avoids making an instance’s loss depend on how many labels are available, and repeat factors depend on label rarity.
  • Supervised contrastive learning with negative-label expansion: The SCoNE framework extends a strong attention-based baseline with negative-label expansion and supervised contrastive learning to learn more attribute-discriminative features.The negative-label scheme adds 5.9M training negatives, while attribute-specific projections pull samples sharing an attribute together.

5. Experiments

On VAW, SB improves over competing baselines, while SB+SCoNE substantially advances performance across metrics and attribute categories. Ablations show additive gains from the proposed components, and qualitative results illustrate accurate localization and complementary attention.

  • 5.1. Experimental setup: The strong baseline combines image-object composition, multi-attention, and object localization, while experiments evaluate mAP, mR@15, and F1 metrics.The evaluation compares these models with multi-label learning and attribute-prediction baselines using ResNet-50 backbones.
  • 5.3. Results: SB outperforms other baselines on almost all metrics, indicating that object localization and multi-attention benefit attribute prediction.ResNet-Baseline-CE has higher mR@15 but much lower precision-oriented mAP and F1.
  • 5.3. Results: SB+SCoNE substantially improves over SB across all metrics and surpasses available algorithms by a large margin.It gains 5 mAP points on long-tail attributes and nearly 7–8 mAP points on color and material attributes over the next-best method.
  • 5.4. Ablation studies: SCoNE components additively improve SB by 2.4 mAP, 5.4 mR@15, and 5 F1@15 points, with negative label expansion providing the largest mAP gain.Applying SCoNE to ResNet-Baseline also improves mAP and mR@15 by 3.4 and 4.7 points, respectively.
  • 5.5. Qualitative results: Qualitative examples show SB+SCoNE predicting diverse attributes while localizing objects and using attention on relevant regions outside the object region.For example, attention to a skateboard helps predict the person’s skateboarding attribute.

6. Discussion, Future works and Conclusion

The paper presents VAW as a large-scale benchmark for in-the-wild attribute prediction and positions SCoNE as improving performance over the strong baseline. It also identifies persistent long-tail imbalance and potential object bias as open challenges.

  • Conclusion: SCoNE improves over the strong baseline by 2.4 mAP and 5.4 mR@15 points on VAW.The paper attributes these gains to its proposed algorithmic improvements.
  • Future works: Tail-category mAP still trails head-category mAP by more than 25 points despite reweighting and resampling improvements.The paper identifies data imbalance as a substantial challenge for future work.
  • Future works: Using object labels is crucial for strong VAW performance but may introduce object bias, motivating evaluation on compositionally novel instances.The paper leaves this investigation outside its current scope.
  • Conclusion: VAW is proposed as a benchmark for attribute prediction in the wild and as a broader test of long-tailed multi-label prediction with limited labels and bias issues.The conclusion also mentions out-of-distribution testing as part of this broader scope.
  • Qualitative analysis: Qualitative results indicate that the object localizer can alleviate occlusion, while multi-attention can use informative regions beyond the object.The described visualizations include object regions, attention maps, and attribute predictions.

B. Additional details for the VAW dataset

The VAW dataset combines positive and negative attribute annotations with rule-based negative-label expansion and exhibits strongly long-tailed category distributions. Despite this imbalance, the models perform similarly across balanced and skewed categories.

  • Dataset annotations: VAW includes positive and negative annotations, plus rule-based negative labels derived from mutually exclusive attributes such as empty and filled.A positive empty annotation can automatically produce a negative filled annotation for the same object.
  • Attribute distribution: The top two material attributes account for over 30.91% of all material annotations, demonstrating strong long-tailed skew.Metal and wooden comprise 41.4% of positive and 23.5% of negative material annotations.
  • Attribute distribution: The strong baseline and SCoNE model perform almost equally well on balanced texture and skewed material categories.This comparison accompanies the dataset’s observed category imbalance.

C.1. Study of different reweighting and resampling methods

The study compares reweighting and resampling methods under VAW’s severe class imbalance, finding that combined RW-BCE and RFS performs best across most metrics.

  • The experiments apply each technique to the ResNet-Baseline trained after negative label expansion.
  • CAS obtains low mAP but decent mR@15 and mA because uniform class sampling can severely undersample head classes and oversample tail classes.Its balanced batches also prevent negative examples from dominating positive examples.
  • IF and CB improve over the baseline across all metrics, while RW-BCE further improves performance by separately weighting positive and negative labels.RW-BCE also highlights rare classes using weighting similar to IF.
  • RW-BCE combined with RFS achieves better results across most metrics than the other evaluated methods.The combined method is referred to as RR and is used in the main paper.

C.2. Components of the Strong Baseline

The ablations show that low-level features, multi-attention, and object localization collectively strengthen the baseline, while jointly trained SupCon avoids the incompatibility seen with attention during pretraining.

  • Strong Baseline components: Removing low-level features lowers mAP for both low-level attributes such as color and material and higher-level attributes such as action.The model otherwise must use a single feature representation for both levels.
  • Strong Baseline components: Removing all attention components severely hampers performance across attribute categories, showing that the full Strong Baseline’s components are collectively important.Removing any individual sub-component also negatively affects performance overall.
  • Strong Baseline components: Removing the object localizer does not drastically reduce performance because weakly supervised multi-attention can still distinguish objects from background and distractors.
  • SupCon and attention: SupCon pretraining improves the Strong Baseline without multi-attention but drops mAP when combined with the attention module.The authors attribute this incompatibility to global average pooling emphasizing foreground context while multi-attention models features across locations.
  • SupCon and attention: Joint SupCon training removes the pretraining problem, improves almost all overall metrics, and provides greater benefits for tail classes.
  • SupCon and attention: The paper notes that supervised contrastive learning remains insufficiently explored for multi-label settings such as VAW.

D. Evaluation metrics

The evaluation uses mAP as its primary ranking metric alongside mR@15, mA, and F1@15, while accounting for VAW’s partially labeled annotations.

  • mAP measures how well each attribute label ranks correct images above incorrect ones and is the primary evaluation metric.Average precision is computed per attribute class.
  • mR@15 measures retrieval of ground-truth positive attributes among each image’s top 15 predictions.Predictions for unannotated labels are ignored because VAW is partially labeled.
  • mA evaluates class-wise accuracy at a 0.5 threshold, so imbalanced positive and negative predictions can reduce its score.The metric uses positive and negative examples for each attribute class.
  • F1@15 is computed from precision and recall using the model’s top 15 predictions, with the paper reporting overall F1 and per-class recall as mR@15.

E. Implementation details

The implementation combines a pretrained ResNet-50 with low-level feature maps, fixed GloVe object embeddings, ontology-derived attribute relations, and separate training procedures for SupCon variants.

  • The model uses ImageNet-pretrained ResNet-50 features from blocks 2 and 3 as low-level representations and fixed 100-dimensional GloVe object embeddings.The embeddings remain frozen to support generalization to unseen objects at test time.
  • Training uses Adam with batch size 64, weight decay 1e-5, separate learning rates for pretrained and remaining parameters, 12 epochs, and validation-based decay.Inputs are 224x224 images with basic augmentations around the object bounding box.
  • SupCon pretraining uses 10 epochs, batch size 384 with 768 views per batch, temperature τ = 0.25, and identity initialization for the A_c matrices.
  • Joint SupCon training adds λ_supL_sup with λ_sup = 0.5 while retaining the other Strong Baseline hyperparameters.
  • Attribute categories and relations are constructed from WordNet, ConceptNetAPI, and a related ontology, then manually verified for categories.Overlapping relations use synonym-like links, while exclusive relations use antonym and distinctness links.

G. Image search results from our SCoNE model

SB+SCoNE retrieves images exhibiting specified visual attributes, including multiple colors, and distinguishes size differences within the same object categories.

  • SB+SCoNE retrieves images exhibiting one or multiple requested attributes, including simultaneous searches for multiple colors.The model’s ranking results cover single-attribute and multi-attribute queries.
  • The retrieved results distinguish small from large instances within the same object categories, including birds, planes, and phones.Figure 11 deliberately compares identical categories across rows to illustrate size discrimination.
  • The image-search evaluation includes queries for color, material, shape, and size attributes.Figures 7–11 present ranking results across these attribute types.
Loading 2106.09707v1…