PaperlayerView exact revision on arXiv

Source-linked AI summary

DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations

Ximeng Sun, Ping Hu, Kate Saenko

arXiv:2206.09541v1cs.CV

TL;DR

Limited multi-label annotations make partial-label and zero-shot recognition difficult, motivating methods that transfer knowledge from pretrained image-text alignment. DualCoOp learns positive and negative class prompts with lightweight adaptation, and experiments report improvements across both low-label settings.

  • Problem

    Partial-label and zero-shot multi-label recognition must recognize multiple categories despite incomplete annotations or unseen test classes.

  • Method

    DualCoOp learns positive and negative contextual prompts around each class name while adapting a pretrained vision-language model with lightweight learnable parameters.

  • Results

    DualCoOp achieves the best performance across tested label proportions and improves mAP by 6.8% on VOC2007 with 10% of labels, while improving NUS-WIDE Top-3 F1-score by 10.8%.

  • Takeaways & Limitations

    The framework provides a unified approach that quickly adapts pretrained vision-language knowledge to partial-label and zero-shot multi-label recognition.

  • Takeaways & Limitations

    The Transformer-based language model limits label-set size, and zero-shot unseen classes still perform worse than models trained with seen and unseen labels.

Abstract

from arXiv · show

Solving multi-label recognition (MLR) for images in the low-label regime is a challenging task with many real-world applications. Recent work learns an alignment between textual and visual spaces to compensate for insufficient image labels, but loses accuracy because of the limited amount of available MLR annotations. In this work, we utilize the strong alignment of textual and visual features pretrained with millions of auxiliary image-text pairs and propose Dual Context Optimization (DualCoOp) as a unified framework for partial-label MLR and zero-shot MLR. DualCoOp encodes positive and negative contexts with class names as part of the linguistic input (i.e. prompts). Since DualCoOp only introduces a very light learnable overhead upon the pretrained vision-language framework, it can quickly adapt to multi-label recognition tasks that have limited annotations and even unseen classes. Experiments on standard multi-label recognition benchmarks across two challenging low-label settings demonstrate the advantages of our approach over state-of-the-art methods.

1 Introduction

Multi-label recognition seeks all semantic labels in complex images, but limited annotations make training difficult. DualCoOp addresses partial-label and zero-shot MLR with positive and negative prompts over pretrained vision-language features.

  • Multi-label recognition identifies all semantic labels present in an image, unlike single-label recognition.
  • Partial-label MLR reduces annotation burden by recording only some categories per training image, while zero-shot MLR recognizes novel categories at test time.
  • Existing approaches model visual-textual alignment and label correlations using limited annotations, producing sub-optimal performance and complex designs.
  • DualCoOp jointly handles both settings by learning positive and negative prompts while relying on large-scale pretrained vision-language alignment.
  • 6.8% mAP improvement with 10% of labels on VOC2007 and 10.8% Top-3 F1-score improvement on NUS-WIDE demonstrate the reported advantages.

2 Related Works

Related work addresses limited-annotation MLR through label modeling, semi-supervision, normalized objectives, embedding spaces, and prompt learning. These methods often require substantial MLR data or complex architectures, motivating DualCoOp’s use of pretrained image-text knowledge.

  • Conventional MLR methods learn category classifiers while incorporating label correlations through graph neural networks or recurrent models.
  • Partial-label methods use web-generated pseudo-labels, annotation proportions, or normalized BCE objectives, but web supervision is noisy and incomplete.
  • Zero-shot MLR transfers knowledge to unseen categories through joint visual-label embedding spaces and specialized ranking objectives.
  • Existing approaches still require substantial MLR data and complex architectures or losses, whereas DualCoOp leverages unsupervised text-image pretraining to reduce data requirements.
  • Vision-language models such as CLIP provide generic visual representations and transfer capability, motivating prompt learning for downstream adaptation.

3 Method

DualCoOp adapts pretrained vision-language encoders to partial-label and zero-shot multi-label recognition with learnable positive and negative prompts, class-specific spatial aggregation, and frozen network components. Its prediction compares positive and negative class evidence after aggregating region-level visual-text similarities.

  • Problem settings: DualCoOp defines partial-label MLR with labels marked positive, negative, or unknown, and zero-shot MLR with seen or unseen categories.The framework predicts each label of interest during inference.
  • Class-Specific Region Feature Aggregation: Class-Specific Region Feature Aggregation projects each region into textual space and weights spatial logits by class-specific semantic-response magnitude.This modifies CLIP’s spatial aggregation to retain region-level information for multiple objects without introducing new aggregation parameters.
  • Inference: During inference, DualCoOp compares the final positive and negative logits to decide whether each class is present.The paired classifiers avoid hand-crafted thresholding for positive-label decisions.
  • Dual Learnable Prompts: DualCoOp uses paired positive and negative prompts, with class-specific pairs for partial-label MLR and shared pairs for zero-shot MLR.Each prompt combines learnable context tokens with a category name and produces a textual embedding through the pretrained text encoder.
  • Prediction: The binary output p compares image similarity with positive and negative textual embeddings using the visual and textual encoders.The aggregation function A reduces spatial visual features for each class before similarity computation.
  • Optimization: During training, ASL updates only the learnable prompts while the pretrained visual and textual network components remain frozen.ASL down-weights and hard-thresholds easy negative samples through its negative-example margin and exponents.

4 Experiments

Experiments evaluate DualCoOp on partial-label and zero-shot multi-label recognition across MS-COCO, VOC2007, and NUS-WIDE, including ablations of text supervision and visual aggregation. DualCoOp achieves strong performance with a small learnable overhead across these settings.

  • Experimental Setup: Experiments cover partial-label recognition on MS-COCO and VOC2007, and zero-shot recognition on MS-COCO and NUS-WIDE.Partial labels are varied from 10% to 90%; zero-shot evaluation includes both unseen-only and generalized settings.
  • Partial-Label Results: DualCoOp achieves the best partial-label performance across all label proportions, with 1.3M versus 29.6M learnable parameters on MS-COCO and 0.3M versus 29.6M on VOC2007.Compared with the second-best method, improvements at 10% labels are 3.2% on MS-COCO and 6.8% on VOC2007.
  • Zero-Shot Results: DualCoOp achieves the best F1 score in all zero-shot and generalized zero-shot cases, using a 0.02M learnable overhead.Zero-shot F1 improves by 12.5 at Top-3 on MS-COCO and by 10.8 at Top-3 and 10.9 at Top-5 on NUS-WIDE.
  • Text Supervision: With 10% of labels, DualCoOp achieves 7.8% higher mAP than Discrete Labels, whereas SST is 1.5% lower than Discrete Labels.The comparison uses the same CLIP initialization and examines the effect of textual supervision.
  • Visual Aggregation: Class-specific feature aggregation benefits from increased input resolution during training or inference, whereas multi-headed attention is tied to CLIP’s 224 resolution.The class-specific aggregation uses original weights and performs better than fine-tuning the original multi-headed attention layer.

5 Conclusion

DualCoOp is presented as a lightweight unified framework for partial-label and zero-shot multi-label recognition, using pretrained vision-language representations and positive/negative prompts. The paper also identifies label-set scalability and zero-shot unseen-class performance as limitations, alongside broader deployment risks.

  • 5 Conclusion: DualCoOp addresses partial-label and zero-shot multi-label recognition with a lightweight adaptation of large-scale vision-language pretraining.It combines positive and negative prompts with class-specific region feature aggregation.
  • 5 Conclusion: The Transformer-based language model limits the size of the label set because all labels must pass through the text encoder.This limitation follows from the computational role of the pretrained language model.
  • 5 Conclusion: Zero-shot unseen-class performance remains worse than training with both seen and unseen labels, despite 400M auxiliary pretraining samples.The paper presents this result as highlighting the difficulty of zero-shot multi-label recognition.
  • 5 Conclusion: Potential deployment risks include adversarial attacks, data poisoning, dataset bias, poor interpretability, privacy violations, and job losses from increased automation.The paper states that these risks are beyond its scope and require caution in deployment.

A Different Prompt Length

Prompt length affects DualCoOp differently across partial-label and zero-shot multi-label recognition. Class-specific prompts perform well with shorter contexts, whereas class-agnostic zero-shot prompts require longer contexts.

  • A Different Prompt Length: Partial-label MLR performs well with small prompt lengths, particularly N = 8 or 16, when class-specific prompts are learned.The main paper uses N = 16.
  • A Different Prompt Length: Zero-shot MLR requires larger prompt lengths, such as N = 32 or 64, when uniform prompts are shared across classes.The prompt-length behavior differs because zero-shot learning uses class-agnostic prompts.
  • A Different Prompt Length: Figures 4 and 5 compare MLR performance across different prompt lengths for partial-label and zero-shot settings on MS-COCO.The comparisons cover prompt lengths N = 2, 4, 6, 8, 16, 32, and 64.

B Performance on the Full Dataset

On the full MS-COCO multi-label recognition dataset, DualCoOp achieves 85.8% mAP with ResNet-101 at input resolution 448, slightly exceeding ASL under the same setting.

  • B Performance on the Full Dataset: 85.8% mAP is achieved by DualCoOp with ResNet-101 and input resolution 448 on full MS-COCO.This result uses the visual backbone finetuned on the full multi-label recognition dataset.
  • B Performance on the Full Dataset: 85.0% mAP is achieved by ASL with the same ResNet-101 and input-resolution setting.The comparison uses the same reported backbone and resolution.
  • B Performance on the Full Dataset: DualCoOp exceeds ASL by 0.8 percentage points under the matched full-dataset setting.The difference is computed from the reported 85.8% and 85.0% mAP values.

C Full performance of MLR with Partial Labels

The supplementary material reports class-averaged and overall precision, recall, and F1 measures for DualCoOp in partial-label MLR on MS-COCO and VOC2007.

  • C Full performance of MLR with Partial Labels: The supplementary results report average per-class precision, recall, and F1 scores for partial-label MLR.These are denoted CP, CR, and CF1.
  • C Full performance of MLR with Partial Labels: The supplementary results also report average overall precision, recall, and F1 scores for partial-label MLR.These are denoted OP, oR, and OF1.
  • C Full performance of MLR with Partial Labels: The reported supplementary metrics cover MS-COCO and VOC2007 and supplement Table 1 in the main paper.The results are provided in supplementary Tables 7 and 8.

D Visualization of Class-Specific Region Feature Aggregation

The paper visualizes DualCoOp’s class-specific region feature aggregation on MS-COCO. The visualization indicates that DualCoOp assigns high attention scores to the correct objects.

  • The visualization evaluates class-specific region feature aggregation on the MS-COCO dataset.
  • DualCoOp produces high attention scores for the correct objects.
  • Figure 6 presents the class-specific region feature aggregation visualization.
Loading 2206.09541v1…