PaperlayerView exact revision on arXiv

Source-linked AI summary

Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Framework

Shu Zhang, Ran Xu, Caiming Xiong, Chetan Ramaiah

arXiv:2204.13207v1cs.CVcs.AIcs.LG

TL;DR

Existing contrastive learning frameworks typically use a single supervisory signal, while hierarchical label relationships receive little attention in representation learning. This paper introduces a multi-label contrastive framework with hierarchy-preserving losses, and reports that its embeddings generalize across downstream tasks and unseen data while outperforming evaluated baselines.

  • Problem

    Representation learning often uses a single supervisory signal and gives little attention to hierarchical label relationships, despite the need for embeddings that generalize across downstream tasks and unseen data.

    Source: §1 · p. 1

  • Method

    The framework uses all available hierarchical labels with HiMulCon’s proximity-dependent positive-pair penalty and HiConE’s hierarchy constraint.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5

  • Results

    The approach generalizes across downstream tasks and unseen data and significantly outperforms the evaluated baselines.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The framework is downstream-task agnostic and can be used to learn embeddings for a variety of downstream tasks.

    Source: §1 · p. 2, §2 · p. 3

    Learning the encoder requires hierarchical labels that can be expensive to acquire, and experiments were tested on tree-like label structures.

    Source: §5 · p. 8

Abstract

from arXiv · show

Current contrastive learning frameworks focus on leveraging a single supervisory signal to learn representations, which limits the efficacy on unseen data and downstream tasks. In this paper, we present a hierarchical multi-label representation learning framework that can leverage all available labels and preserve the hierarchical relationship between classes. We introduce novel hierarchy preserving losses, which jointly apply a hierarchical penalty to the contrastive loss, and enforce the hierarchy constraint. The loss function is data driven and automatically adapts to arbitrary multi-label structures. Experiments on several datasets show that our relationship-preserving embedding performs well on a variety of tasks and outperform the baseline supervised and self-supervised approaches. Code is available at https://github.com/salesforce/hierarchicalContrastiveLearning.

1. Introduction

Hierarchical multi-label data is common, but representation learning often treats labels as flat or uses only one supervisory signal. The paper proposes contrastive losses that use all label levels to preserve their hierarchy in embeddings and reports performance across downstream tasks and unseen data.

1. Introduction

  • Flat-label learning leaves hierarchical relationships underused, although preserving them in embeddings matters when one representation must serve multiple downstream tasks.The introduction contrasts flat class lists with representation learning, where the embedding must generalize to unknown tasks and unseen data.Source: §1 · p. 1
  • Prior contrastive approaches treat positive pairs equally and do not support multi-label relationships, limiting direct supervision for retrieval, attributes, and clustering.Self-supervised methods use augmented views; supervised methods add same-class pairs, but both minimize positive-pair distances to the same degree.Source: §1 · p. 1
  • In hierarchical multi-label data, each sample has multiple dependent labels organized in a hierarchy, as in DeepFashion’s category, product, and variation labels.The anchor’s same-product image is a sub-category positive, while an image from another product in the same category is a category-level positive.Source: §1 · p. 1
  • HiMulCon weights positive-pair penalties by label-space proximity, while HiConE enforces that farther-apart pairs cannot have lower loss than closer pairs.Proximity is defined by shared ancestry in the label tree; the framework can combine the two losses.Source: §1 · p. 2
  • The framework reduces to supervised contrastive learning with single-level labels and SimCLR [5] without labels, and is reported to preserve hierarchy and generalize to unseen data.The introduction reports evaluations on category prediction, sub-category retrieval, and clustering NMI.Source: §1 · p. 2

2. Related Work

Related work spans self-supervised and supervised contrastive learning, hierarchical classification, and metric-learning approaches. The paper distinguishes its framework by learning from natural multi-level labels for downstream-agnostic representations rather than optimizing a particular task.

2. Related Work

  • Self-supervised contrastive learning forms positives from different views of one image, while supervised contrastive learning samples positives from different instances of the same class [5].Supervised contrastive learning generalizes triplet and n-pair loss objectives.Source: §2 · pp. 2–3
  • Prior hierarchical methods include multi-label contrastive learning based on any shared label and approaches for classification, hierarchical constraints, or structured retrieval [22] [12] [39] [19].These approaches are distinguished from the proposed method, which builds the hierarchy on natural data characteristics.Source: §2 · p. 3
  • Ge et al. [11] construct a tree from single-level labels and intraclass distances for hierarchical triplet learning, whereas this approach uses multi-level labels for contrastive learning.The paper notes that data-derived trees can propagate model and data biases and be prone to noise.Source: §2 · p. 3
  • Unlike multi-task methods [12], the proposed framework is downstream-task agnostic and is not directly optimized for classification or retrieval.Those tasks are used in experiments, but the framework is presented as applicable to any downstream task.Source: §2 · p. 3

3. Methodology

The methodology builds representations from hierarchical labels and defines losses that preserve label relationships across levels. Its batch sampler supplies positive pairs sharing ancestry at every level.

3. Methodology

  • A hierarchical multi-label dataset assigns each data point multiple related labels, represented by a tree or directed acyclic graph; the method learns an encoder and freezes it for downstream tasks.A separate classifier can use the frozen encoder's embeddings for downstream classification.Source: §3 · pp. 3–4

3.1. Contrastive Learning

  • Contrastive learning pulls an anchor toward positive samples and separates it from negatives; supervised contrastive learning extends this to multiple labeled positives.The contrastive objective compares the anchor-positive similarity with similarities to other batch samples.Source: §3.1 · p. 4

3.2. Hierarchical Multi-label Contrastive Loss

  • HiMulCon applies level-specific penalties to positive pairs, making higher-level pairs closer while lower-level pairings provide hard negatives; with one label level, it reduces to supervised contrastive loss.The level penalty explicitly contributes to preserving the hierarchy.Source: §3.2 · p. 4

3.3. Hierarchical Constraint Enforcing Loss

  • HiConE enforces hierarchy by requiring pair losses at higher levels not to exceed those at lower levels, computing levels sequentially to maintain that constraint.The computation proceeds in decreasing order of level.Source: §3.3 · p. 5

3.4. Hierarchical Multi-label Constraint Enforcing Contrastive Loss

  • HiMulConE combines a level-weighted penalty with a hierarchy constraint that relates pair losses across levels.The level penalty is independent, while the constraint is dependent on losses at other levels.Source: §3.4 · p. 5

3.5. Hierarchical Batch Sampling Strategy

  • The sampler selects images sharing common ancestry with an anchor at each label level, ensuring batches contain positive pairs across the hierarchy.Images are sampled from the relevant label subtrees, with each image sampled only once per epoch.Source: §3.5 · p. 5

4. Experiments

The experiments assess classification, retrieval, and clustering on standard datasets and unseen splits, reporting results that favor the proposed approach across these tasks.

4.1. Datasets

  • Experiments use standard dataset configurations and also create seen/unseen splits for some datasets to test generalization.The datasets include ImageNet, DeepFashion In-Shop, iNaturalist, and ModelNet40; DeepFashion In-Shop has category, product-ID, and variation labels.Source: §4.1 · pp. 5–6

4.4. Image Retrieval Accuracy

  • The proposed losses outperform retrieval baselines on full datasets, with HiMulConE showing greater improvement at smaller k.The comparison includes triplet loss and FashionNet alongside the classification-experiment baselines.Source: Figure 3 · p. 6, §4.4 · p. 7

4.2. Implementation Details

  • Training uses a ResNet-50 encoder, with the fourth layer and an MLP head finetuned on seen data before encoder weights are frozen.A linear classifier is then trained for 40 epochs; the reported setup uses batch size 512 and temperature 0.1.Source: §4.2 · p. 6

4.5.2 Classification

  • On unseen DeepFashion and ModelNet40, proposed methods beat baselines while performing comparably to SupCon on seen data; HiConE gains more with semantic overlap.HiConE does better on DeepFashion, where semantic overlap across hierarchy levels is greater; the gap is smaller on ModelNet.Source: Table 2 · p. 7, §4.5.2 · p. 8

4.3. Classification Accuracy

  • The classification comparison includes cross entropy, SimCLR, SupCon, and Guided; all approaches use the same classifier-training mechanism.Supervised approaches are competitive, and the proposed approach uses additional labels during representation learning that are not used to train the classifier.Source: §4.3 · p. 7

4.5.4 Clustering

  • Product NMI improves significantly over baseline, indicating within-category subcategory separability and preservation of the label hierarchy.Clustering is evaluated at category and product-ID levels on unseen test embeddings.Source: Table 3 · p. 7, §4.5.4 · p. 8

4.5.3 Image Retrieval

  • On unseen DeepFashion and ModelNet splits, the losses outperform baselines, particularly at lower k, supporting generalization to unseen data.The authors connect incorporating label hierarchy in the loss with preservation of label-space hierarchy in the embedding space.Source: Figure 4 · p. 8, §4.5.3 · p. 8

5. Limitations and Social Impact

The approach requires hierarchical labels for encoder learning, is tested on tree-structured labels, and may learn biases present in the data.

5. Limitations and Social Impact

  • Encoder learning requires hierarchical labels, which can be expensive to acquire.This constrains use where hierarchical annotations are unavailable or costly.Source: §5 · p. 8
  • Experiments use tree-like label structures in which each node has one parent; the authors describe extension to general graphs as relatively straightforward.This marks the tested structure, not a demonstrated evaluation on general graphs.Source: §5 · p. 8
  • The model can learn biases in the underlying data, which may have significant societal impact; the authors call for debiasing through re-annotation or dataset restructuring.Source: §5 · p. 8

6. Conclusion

The conclusion presents a framework for using available labels in hierarchical data, with standard supervised or self-supervised approaches as special cases when data are insufficient.

6. Conclusion

  • The framework uses available labels for hierarchically categorized data and reduces to standard supervised or self-supervised approaches when sufficient labels are absent.The authors identify extending to non-hierarchical multi-label scenarios and other modalities as future work.Source: §6 · p. 8

Framework: Supplemental Material

The supplied passage identifies the paper as an arXiv preprint dated 27 April 2022.

Framework: Supplemental Material

A. Details of Seen and Unseen Data

The section describes the seen/unseen data splits and evaluation setups for DeepFashion and ModelNet40.

A. Details of Seen and Unseen Data

  • DeepFashion has 25,900 training, 12,612 validation, and 14,218 test images; query images serve as test images for category classification.The model is trained on seen categories, then its classifier is finetuned on unseen categories; train and test images and seen and unseen categories do not overlap.Source: A. Details of Seen and Unseen Data · p. 11
  • DeepFashion retrieval uses header features to compare query and gallery images, while classification tests transfer to unseen categories.The test images are queries for category classification, and the retrieval setup uses no overlap in categories between seen and unseen data.Source: A. Details of Seen and Unseen Data · p. 11
  • ModelNet40 contains 3,183 CAD models from 40 classes, split into 22 seen and 18 unseen categories.Its retrieval gallery has 11,221 images and its query set has 6,017; because no official retrieval split exists, the authors use a split similar to DeepFashion In-Shop’s validation/test ratio.Source: A. Details of Seen and Unseen Data · p. 11

B. Qualitative assessment of Image Retrieval

Qualitative retrieval examples compare the proposed losses with baselines on unseen data, showing differences in product-ID retrieval and category matches.

on Unseen data

  • In Figure 1(a), the top two retrieved images from each of the three proposed algorithms are correct, while SimCLR and cross-entropy retrieve no correct images.Most baseline results nevertheless belong to the correct categories.Source: on Unseen data · p. 11
  • In Figure 1(c), only HiMulConE retrieves the correct product ID; the other methods, including the two individual proposed losses, fail.The authors describe this as the most challenging example because the query contains both tees and denims.Source: on Unseen data · p. 11
  • In Figure 1(b), HiMulCon and HiMulConE retrieve correct images in their top three, while SimCLR’s four category matches have incorrect product IDs.The passage says the proposed methods’ retrieved images are correct at the product level, whereas SimCLR’s category matches are not correct products.Source: on Unseen data · p. 11
  • Across the examples, the proposed losses often return tee-denim combinations, which the authors regard as reasonable context for the query images.They argue that the combined loss gives the best learning ability and good separability at both category and sub-category levels.Source: on Unseen data · p. 11

C.1. Sampling Strategy

The appendix examines hierarchical batch sampling, penalty choices in HiMulCon, and embedding visualizations on DeepFashion and ModelNet40.

C.1. Sampling Strategy

  • With HiConE and fixed hyperparameters, completely random sampling significantly worsens category prediction; hierarchical sampling targets positive pairs at every level.The hierarchical strategy also makes lower-level pairs hard samples for higher levels, and random sampling produced no non-augmentation positive pairs in over 20% of batches.Source: §C.1 · pp. 11–12, Table 1 · p. 12

C.2. Layer Penalty in HiMulCon

  • The directly proportional λ_l candidate was selected for other experiments, while inversely proportional functions performed worse than those in the table.The design intuition is to force higher-level pairs closer than lower-level pairs.Source: §C.2 · p. 12

C.3. Layer Penalty in HiMulCon: Sanity check ex-

  • Identity penalizes all layers equally and makes HiMulCon approximately equal to SupCon, while HiMulConE retains a hierarchy benefit; decreasing exponential penalties significantly deteriorate performance.The sanity checks compare these inverse-proportional functions with the directly proportional penalty choices.Source: §C.3 · pp. 12–13, Table 2 · p. 13

D. t-sne visualization

  • On DeepFashion, the proposed losses show category-level separability, with semantically similar categories closer than unrelated ones; on ModelNet40, they show good separability.SimCLR and SupCon clusters have significant category mixing on DeepFashion, and their ModelNet40 subspaces correlate less with category labels.Source: D. t-sne visualization · p. 13

E. Hierarchy Constraint Study

The study tracks hierarchical constraint violations during training and on held-out test data, finding that the proposed losses reduce violations.

E. Hierarchy Constraint Study

  • During training, all three losses reduce hierarchy-constraint violations, but HiMulCon does worse than the other two; HiMulConE’s penalty reduces violations more than HiMulCon.The study tracks violations across batches before calculating the loss, aggregated by epoch.Source: E. Hierarchy Constraint Study · p. 13
  • On held-out test data, the authors track violations by sampling image pairs, finding their lowest common ancestor, and comparing pair distances in the embedding space.A violation occurs when a higher-level pair is closer than a lower-level pair.Source: E. Hierarchy Constraint Study · pp. 13–14
  • The authors present the test-set violation results as additional evidence that their framework preserves hierarchical label structure in the embedding space.The table caption states that lower numbers indicate fewer pairs violating the hierarchy constraint.Source: Table 4 · p. 13, E. Hierarchy Constraint Study · p. 14
Loading 2204.13207v1…