PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapting Dense Vision-Language Relationships for Multi-label Classification with Partial Label

Cheng Chen, Yifan Zhao, Jia Li

arXiv:2608.22313v1cs.CV

TL;DR

Partial-label multi-label classification must recover missing semantics without relying on unstable statistical priors that can cause catastrophic overfitting. LDSA adapts pretrained CLIP knowledge through dense visual contrastive learning and language-driven interactions, achieving state-of-the-art benchmark performance.

  • Problem

    Partial-label multi-label classification needs missing semantics recovered, while statistical priors can make unstable semantic mistakes and cause catastrophic overfitting.

  • Method

    LDSA adapts pretrained multimodal CLIP knowledge using a densely contrastive visual adaptor and a language-driven interactive decoder with class-specific prompt tuning.

  • Results

    LDSA achieves state-of-the-art performance on MS-COCO, PASCAL VOC, and Visual Genome, improving over state-of-the-art methods by over 2.8%, 9.3%, and 0.5%, respectively.

  • Takeaways & Limitations

    Jointly adapting visual and language representations enables prior-adaptive generalized knowledge for partial multi-label learning and discovers implicit semantic relationships.

  • Takeaways & Limitations

    Class-specific specialization may limit open-set generalization, and the theoretical understanding of the method’s contrastive effects remains preliminary.

Abstract

from arXiv · show

Learning multi-label image classification with incomplete annotations is a challenging task that has been widely studied for its superior trade-off between high efficiency and less labor consumption on large-scale datasets. Predominant methods rely on strong prior assumptions to recover the missing semantics from partial annotations. However, these statistic priors suffer from unstable semantic mistakes and thus lead to catastrophic overfitting. Toward this end, we propose a Language-driven Dense Semantic Adaptor (LDSA) that excavates prior-adaptive relationships from multimodal pretrained CLIP models. In our approach, the densely contrastive adaptor is first proposed to construct dense visual contrastive constraints, transferring the task-specific knowledge to visual domains. We then propose a language-driven interactive decoder with the help of class-specific prompt tuning, which adapts language proxies with visual domains. With the collaborative learning of proposed modules, experimental results demonstrate our proposed LDSA achieves a new state of the art on public multi-label classification benchmarks, and interpretable analyses reveal that our LDSA discovers implicit semantic relationships with the prior-adaptive learning scheme.

1 INTRODUCTION

Partial-label multi-label classification reduces annotation demands but leaves missing semantics to recover. LDSA adapts generalized CLIP knowledge through dense visual contrastive learning and language-driven interaction, outperforming prevailing methods across benchmarks.

  • Partial-label learning addresses the cost of constructing large-scale multi-label datasets but requires recovering missing labels from incomplete annotations.
  • Existing methods rely on statistical or language-based priors, whose dependence on training-data statistics can produce semantic mistakes and limit adaptation across downstream scenarios.
  • Its densely contrastive adaptor preserves spatially dense visual features and forms contrastive relationships across augmented views to extract task-specific visual relations.
  • A language-driven interactive decoder uses class-specific prompt tuning to adapt language prototypes to visual domains and refine dense visual features through cross-modal interactions.
  • 2.8% on MS-COCO, 9.3% on Visual Genome, and 0.5% on PASCAL VOC: LDSA improves over prevailing methods, with fast convergence reported.
  • LDSA excavates prior-adaptive relationships from pretrained multimodal CLIPs instead of relying on fixed prior assumptions.

2 RELATED WORKS

Related work spans multi-label recognition, vision-language transfer, and contrastive learning. The paper builds on CLIP’s aligned semantic knowledge and uses batch normalization to support implicit contrastive learning on frozen visual encoders.

  • Multi-label recognition methods model label dependencies, diverse co-occurrence, and spatial-semantic correlations to improve prediction of multiple objects.
  • The LDSA framework combines dense visual representation constraints with language-guided transfer of text priors to address partial-label multi-label learning.
  • Vision-language models align visual and textual features, while CLIP transfers generalized semantic structures to downstream tasks through parameter-efficient adaptation.
  • Contrastive learning distinguishes instances, but multi-label images violate the single-salient-object semantic-consistency assumption used by many conventional contrastive approaches.
  • Batch normalization stabilizes layer-output distributions and is exploited here to accomplish implicit contrastive learning on frozen CLIP features.

3 APPROACH

LDSA adapts pretrained CLIP for partial-label multi-label classification by preserving dense visual representations while restoring vision-language alignment. Its modules jointly impose contrastive consistency, language-guided interaction, and compact semantic prediction.

  • 3.1 Formulations and Framework: Partial-label learning uses incompletely specified binary category vectors, leaving most labels unknown for each image.
  • 3.1 Formulations and Framework: LDSA adapts CLIP with visual features, text priors, and a guidance module to construct downstream predictions.
  • 3.2 Implicit Contrastive Learning: Dense features retain spatial representations of different objects, enabling object-relation discovery and multimodal interactions beyond pooled CLIP vectors.
  • 3.2 Implicit Contrastive Learning: The densely contrastive adaptor combines dense feature regularizers and transformer encoders with residual connections to align strong and weak image views.
  • 3.2 Implicit Contrastive Learning: Naive CLIP fine-tuning achieves 34.0 mAP on MS-COCO, motivating a frozen visual encoder with a non-invasive adaptor to avoid feature collapse.
  • 3.2 Implicit Contrastive Learning: Joint statistics for dual augmentation views encourage a compact semantic space with positive patch pairs within images and negative pairs across images.
  • 3.3.2 Cross-modal Interactive Decoding: Prompt learning and cross-modal decoding realign text prototypes with dense visual features, while compact prediction uses C × D × 1 rather than CD × C parameters.
  • 3.4 Implicit Semantic Contrastive Learning: Feature-space analysis shows that combining visual adaptation with language-guided learning preserves image correspondence and semantic prototype relationships.

4 EXPERIMENTS

LDSA is evaluated on three partially labeled multi-label benchmarks across varying known-label proportions, with comparisons, ablations, and efficiency analyses. It consistently improves benchmark performance, while experiments identify contributions from dense representations, language guidance, augmentations, and feature regularization.

  • Experimental Setup: Experiments use PASCAL VOC 2007, MS-COCO, and VG-200 with 10%–90% known training labels and mAP as the main metric.Partially labeled datasets are generated by randomly dropping labels from fully annotated training data, while validation annotations remain complete.
  • Comparison with State-of-the-art: LDSA surpasses state-of-the-art methods across known-label proportions, improving average mAP by 2.8% on MS-COCO and 9.3% on VG-200.It achieves comparable performance on PASCAL VOC while retaining advantages across varying label proportions.
  • Comparison with State-of-the-art: LDSA balances precision and recall, greatly improving MS-COCO F1 scores and all reported metrics on PASCAL VOC and VG-200.DualCoOp obtains better MS-COCO overall and per-class recall, but with lower precision.
  • Ablation Studies: Dense representations and visual adaptors raise MS-COCO mAP from 34.0 for naively fine-tuned CLIP to 67.3 for the baseline and 73.2 after dense adaptation.The reported progression attributes the later improvement to dense representations coupled with visual adaptors.
  • Ablation Studies: Language guidance produces a relative 10.8% improvement in the ablation, while higher-resolution dense features outperform lower-resolution alternatives.Excessively high resolutions preserve detail but add background noise and computational overhead, motivating a performance-efficiency balance.
  • Ablation Studies: Augmentations improve performance only with the adaptor, whose regularizers stabilize high-variance strong-augmentation features and prevent degradation when components are removed.Additional ablations report that semantic-aware compact predictions outperform alternative solutions with fewer parameters, despite annotation-driven overfitting risks.
  • Efficiency Analysis: Caching language prototypes removes the text encoder at inference, reducing time relative to vanilla CLIP while retaining best performance and balanced resource use.The paper characterizes the resulting method as fast to train and resource-efficient during inference.

5 DISCUSSION AND INTERPRETATION

Interpretation studies examine LDSA’s contrastive behavior, feature distributions, visual attention, learned semantic relationships, and few-shot generalization. They report semantic organization and competitive limited-supervision performance, while acknowledging a possible open-set generalization boundary.

  • How Does Implicit Contrastive Learning Work?: Adding InfoNCE or positive L1 constraints converges faster but reaches similar final results, whereas unconstrained negative training reduces performance from 79.1 to 78.7.The authors conclude that implicit contrastive learning captures positive and negative relationships without requiring additional image-level constraints.
  • Feature Distributions: The adaptor organizes weakly and strongly augmented features into a uniformly distributed space, unlike the clearer decision boundaries observed without it.The visualization uses UMAP on features from 1,000 randomly selected MS-COCO images.
  • Language Prototype Analysis: Adapted language prototypes remain as uniformly distributed as vanilla CLIP while occupying a more distant space, preserving semantic knowledge and generalization.Without the adaptor, prototypes remain distinguishable by distance but lose part of their distributional information.
  • Where does LDSA Focus?: LDSA produces more precise activation maps and attends to object interactions and scene context, whereas the baseline can focus on visually similar objects or background noise.The analysis highlights distinctions such as surfboards versus skateboards through semantic information in surrounding objects and scenes.
  • What Semantic Relationships Do We Learn?: LDSA learns prior-adaptive correlations that include meaningful co-occurrence, semantic meaning borrowing, and coarse semantic relationships beyond fixed co-occurrence or generic CLIP relations.The examples include vase–plotted plant, bear–teddy bear, and carrot–orange relationships.
  • Few-shot Generalization: LDSA achieves the best MS-COCO result in the 1-shot setting and remains competitive in the 5-shot setting.These results are presented as evidence of task-specific adaptation with promising generalization under limited supervision.
  • Limitations and Future Works: Class-specific specialization may limit open-set generalization compared with more generic prompt-learning methods, and the underlying contrastive principles remain only preliminarily explored.The authors identify deeper theoretical understanding and broader-domain generalizability as future directions.

6 CONCLUSIONS

LDSA improves partial multi-label learning by adapting generalized knowledge from visual and language representations to refine prior assumptions. With joint visual-language learning, it achieves state-of-the-art results across three benchmark datasets.

  • LDSA excavates prior-adaptive generalized knowledge through a densely contrastive adaptor and a language-driven interactive decoder.The adaptor targets visual representation, while the decoder transfers generalized language knowledge to visual domains.
  • Joint learning of visual and language representations supports LDSA’s partial multi-label classification approach.
  • LDSA achieves state-of-the-art performance on MS-COCO, PASCAL VOC, and Visual Genome for partial multi-label learning.
Loading 2608.22313v1…