PaperlayerView exact revision on arXiv

Source-linked AI summary

CHiLS: Zero-Shot Image Classification with Hierarchical Label Sets

Zachary Novack, Julian McAuley, Zachary C. Lipton, Saurabh Garg

arXiv:2302.02551v3cs.CVcs.LG

TL;DR

Coarse or uninformative class names leave open-vocabulary zero-shot classification with a gap in how classes are represented. CHiLS predicts over subclasses and maps the result back to a superclass, using existing hierarchies or GPT-3-generated sets. It improves accuracy across both hierarchy settings, with gains up to 30% when ground-truth hierarchies are available.

  • Problem

    Open-vocabulary zero-shot classification uses class names as prompts, but little work has enriched class names themselves, despite coarse labels in datasets with richer underlying structure.

    Source: §1 · p. 1

  • Method

    CHiLS maps classes to existing or generated subclasses, performs zero-shot CLIP prediction across the subclass sets, and maps the predicted subclass back to its superclass.

    Source: §1 · p. 1, §3 · p. 4, §3.1 · p. 4

  • Results

    CHiLS performs as well or better than standard zero-shot practices with synthetic hierarchies and achieves up to 30% accuracy gains when ground-truth hierarchies are available.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For coarse or ill-formed labels, practitioners can use automated GPT-3 subclass generation or hand-crafted label sets without training or fine-tuning.

    Source: §1 · p. 2, §6 · p. 8

    CHiLS is suited to tasks with likely semantic hierarchies, may be less useful for already fine-grained classes, and has no way to validate its performance.

    Source: §6 · p. 8

Abstract

from arXiv · show

Open vocabulary models (e.g. CLIP) have shown strong performance on zero-shot classification through their ability generate embeddings for each class based on their (natural language) names. Prior work has focused on improving the accuracy of these models through prompt engineering or by incorporating a small amount of labeled downstream data (via finetuning). However, there has been little focus on improving the richness of the class names themselves, which can pose issues when class labels are coarsely-defined and are uninformative. We propose Classification with Hierarchical Label Sets (or CHiLS), an alternative strategy for zero-shot classification specifically designed for datasets with implicit semantic hierarchies. CHiLS proceeds in three steps: (i) for each class, produce a set of subclasses, using either existing label hierarchies or by querying GPT-3; (ii) perform the standard zero-shot CLIP procedure as though these subclasses were the labels of interest; (iii) map the predicted subclass back to its parent to produce the final prediction. Across numerous datasets with underlying hierarchical structure, CHiLS leads to improved accuracy in situations both with and without ground-truth hierarchical information. CHiLS is simple to implement within existing zero-shot pipelines and requires no additional training cost. Code is available at: https://github.com/acmi-lab/CHILS.

1. Introduction

CHiLS addresses coarse or uninformative class labels by using hierarchical label sets in zero-shot CLIP classification. Across benchmarks, it improves accuracy with existing hierarchies and generally performs as well as or better than standard zero-shot practice with synthetic hierarchies.

1. Introduction

  • Open-vocabulary models use natural-language class prompts for zero-shot classification, but improving the richness of class names has received little attention despite coarse labels in datasets with richer underlying structure.For example, CIFAR20's “large man-made outdoor things” includes bridges, roads, castles, and skyscrapers.Source: §1 · p. 1
  • CHiLS maps each class to subclasses, predicts across the union of those subclasses, and maps the winning subclass back to its superclass.The method is designed for coarsely defined classes and can use existing or synthetically generated hierarchies.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • CHiLS consistently performs as well or better than standard zero-shot practices with synthetic hierarchies and can achieve up to 30% accuracy gains when ground-truth hierarchies are available.The reported gains apply to image-classification benchmarks with underlying semantic substructure.Source: §1 · p. 2

2. Related Work

Related work explores improving CLIP through few-shot adaptation, zero-shot methods, and hierarchical classification. CHiLS differs by using class hierarchies at inference time for flat superclass prediction without subclass-level supervision.

2.1. Few-Shot Learning with CLIP

  • Prior work improves fine-tuned CLIP capabilities by interpolating fine-tuned and zero-shot weights under distribution shifts or patching them to avoid catastrophic forgetting; separate approaches use adapters or learnable prompts.Zhou et al. (2022a) and Zhou et al. (2022b) optimize learnable prompt tokens using only a few images per class.Source: §2.1 · pp. 2–3

2.2. Zero-Shot Prediction

  • Zero-shot CLIP research spans varied tasks, while image-classification work has explored richer prompts or preexisting captions; CHiLS instead leverages class hierarchy with fixed prompt templates.The surveyed directions include multimodal and generative tasks, and some methods use external knowledge databases.Source: §2.2 · p. 3

2.3. Hierarchical Classification

  • Unlike much hierarchical classification work that assumes training, CHiLS uses class names at inference and predicts flat superclasses without subclass-level supervision.The paper distinguishes this zero-shot, training-free setting from approaches that use supervision or hierarchical information during training.Source: §2.3 · p. 4

3. Proposed Method: CHiLS

CHiLS modifies zero-shot CLIP classification by changing the class set rather than the prompt function. It predicts in subclass space, then reweights subclass probabilities using superclass confidence before returning a superclass prediction.

3. Proposed Method: CHiLS

  • CHiLS modifies the class set when labels are ill-formed or overly general, while keeping the prompt function fixed.Its two main steps are prediction over label subclasses and alignment of subclass probabilities with raw superclass outputs.Source: §3 · p. 4

3.1. Zero-Shot Prediction with Hierarchical Label Sets

  • Each class is mapped to linguistic hyponyms, and CLIP predicts across the union of all subclass sets before the winning subclass is mapped back to its superclass.The paper considers both readily available hierarchies and subclass sets generated by prompting GPT-3.Source: §3.1 · p. 4

3.2. Reweighting Probabilities With Superclass Confidence

  • CHiLS reweights each subclass probability by its superclass probability, down-weighting disagreements when a superclass is confident and relying more on subclasses when superclass probabilities are roughly uniform.The reweighting addresses the loss of superclass information and sensitivity to subclass-label quality in subclass-only prediction.Source: §3.2 · pp. 4–5

4. A Motivating Example for CHiLS

The section tests CHiLS on ImageNet across progressively finer hierarchy levels, using true leaf classes. Gains are strongest at coarse-to-intermediate granularities, where labels resemble intermediate hierarchy nodes.

4. A Motivating Example for CHiLS

  • The ImageNet experiment evaluates zero-shot classification at progressively finer hierarchy levels, providing CHiLS with the leaf nodes for each class at the current level unless the class is itself a leaf.Source: §4 · p. 5
  • As hierarchy depth increases, the gap between CHiLS and standard zero-shot performance decreases while the number of leaf nodes increases.Source: §4 · p. 5
  • The results identify tasks with labels resembling intermediate ImageNet hierarchy nodes as a potential use case for CHiLS.Source: §4 · p. 5
  • CHiLS shows clear zero-shot performance gains over the baseline at coarse-to-intermediate ImageNet hierarchy granularities.The table evaluates settings where CHiLS has access to true ImageNet leaf-node classes.Source: Table 1 · p. 5

5. Experiments

Across the evaluated benchmarks, CHiLS improves on superclass-label baselines with available hierarchies and in most datasets with GPT-3-generated hierarchies. Experiments also examine reweighting, noisy hierarchies, model backbones, and label-set size.

5.1. Setup

  • The 16-benchmark evaluation spans varied image domains and class granularity, compares superclass-label baselines with existing or GPT-3-generated hierarchies, and selects prompts for each dataset to maximize baseline performance.The experiments also examine distribution shifts in BREEDS, Office31, and OfficeHome.Source: §5.1 · pp. 5–6, Table 2 · p. 5

5.2. Leveraging Available Hierarchy Information

  • With available hierarchies, CHiLS improves accuracy across all seven datasets, often by +15%; with GPT-3-generated hierarchies, it improves accuracy in all but two datasets.In the unknown-hierarchy setting, gains are lower, but CHiLS beats the baseline in most datasets.Source: Table 2 · p. 5, §5.2 · p. 6, §5.3 · p. 6

5.3. CHiLS in Unknown Hierarchy Settings

  • Without ground-truth hierarchies, CHiLS beats the baseline in most datasets, though accuracy gains are lower than with true hierarchies.The method uses GPT-3-generated possible subclasses, with adjustments for potentially suboptimal labels.Source: §5.3 · p. 6

5.4. Ablations

  • Without reweighting, performance can improve slightly with true hierarchies but falls below the baseline without them; reweighting combines superclass and subclass predictions.The authors argue that reweighting matters because the ground-truth hierarchy may be unavailable in practice.Source: §5.4 · p. 7, Table 3 · p. 7
  • Using aggregate subclass probabilities for reweighting removes CHiLS’s accuracy gains, while reweighting in superclass space maintains them.The reported upper bound from combining superclass and subclass predictions is only slightly higher than CHiLS.Source: §5.4 · p. 7, Table 4 · p. 7
  • CHiLS improves baseline accuracy even when available hierarchies include extra, out-of-distribution subclasses.The noisy label sets include all ImageNet classes descended from the relevant hierarchy node.Source: §5.4 · p. 7, Table 5 · p. 7
  • Label-set size has little consistent effect across datasets except at extreme sizes, and the proposed aggregation design performed best among tested alternatives.Optimal label-set size depends on the number and semantic granularity of classes.Source: §5.4 · p. 8
  • Across six CLIP backbones, CHiLS shows relatively consistent relative accuracy gains, suggesting its benefits are not an artifact of model scaling.The apparent advantage for ResNet over ViT backbones is slight and not confidently significant.Source: §5.4 · p. 8

6. Conclusion

CHiLS improves zero-shot CLIP accuracy by using hierarchical information, including GPT-3-generated subclasses when explicit subclass information is unavailable. The conclusion highlights its no-training, resource-efficient flexibility and notes boundaries and open questions.

6. Conclusion

  • CHiLS improves zero-shot accuracy with available hierarchical structure and can also improve baseline superclass accuracy using GPT-3-generated subclasses.The method can use existing or generated hierarchies.Source: §6 · p. 8
  • Figure 3 reports only slight variation in CHiLS effectiveness across CLIP backbones and little performance difference across label-set sizes.Source: Figure 3 · p. 8
  • CHiLS can be fully automated with GPT-3 or incorporate hand-crafted label sets, without training or fine-tuning, and is resource efficient.The conclusion presents it as applicable when labels are ill-formed or overly coarse.Source: §6 · p. 8
  • CHiLS may be less useful for classification tasks whose classes are already fine-grained, and its performance cannot be validated within the usual zero-shot setting.The authors say practitioners can decide whether a task likely has a latent hierarchy.Source: §6 · p. 8
  • The authors identify understanding why CHiLS improves accuracy and principled reconciliation of superclass and subclass predictions as future work.Source: §6 · p. 9

A. Empirical Evidence of CLIP Confidence

The appendix reports that CLIP's correct predictions tend to have higher argmax probabilities than its incorrect ones. It also reviews CLIP's zero-shot prompting setup and describes dataset-specific prompt choices, which were not uniformly better than general prompts.

A. Empirical Evidence of CLIP Confidence

B. CLIP Primer

  • CLIP compares image and text embeddings for zero-shot classification, and caption-like prompts can improve prediction over using class names alone.The standard baseline prompt described is “A photo of a {}.”Source: B. CLIP Primer · p. 14

C. Adding Context to Prompts and GPT-3 Queries

  • Dataset-specific prompts were not uniformly better: general ImageNet prompts performed better on Fashion-MNIST and OfficeHome.The experiments used dataset-specific prompt sets for several datasets and added context tokens where applicable.Source: C. Adding Context to Prompts and GPT-3… · pp. 14–15

D. Including Superclass Labels in Label Sets

Adding superclass labels to label sets balances subclass and superclass predictions, but its effect differs between existing and GPT-3-generated maps.

D. Including Superclass Labels in Label Sets

  • Adding superclass labels hurts performance with existing maps but helps maintain performance in most GPT-3-map datasets.The prose says removal of the superclass label hurts in most GPT-3-map cases, sometimes by multiple percentage points, with Fruits360 and Entity30 as exceptions.Source: Table 7 · p. 15, D. Including Superclass Labels in… · p. 15

E. Label Set Ablation Accuracy

The supplied passage identifies a table of CHiLS accuracy scores across label-set sizes.

E. Label Set Ablation Accuracy

F. Alternative Aggregation Methods (Cont.)

The section compares alternative ways to aggregate subclass and prompt embeddings, and describes how richer ImageNet labels can be used to test noisy available hierarchies. The original aggregation choices achieve greater accuracy, while GPT label-set size shows no consistent performance trend across datasets.

F. Alternative Aggregation Methods (Cont.)

  • The original combination of linear prompt averaging and set-based subclass mapping achieves greater average accuracy than the tested alternatives.Linear averaging of subclass embeddings performs worse than set-based mapping; set-based prompt mapping performs worse than linear averaging for superclass prediction.Source: F. Alternative Aggregation Methods… · p. 15, Table 9 · p. 16
  • Across datasets, GPT-3 label-set size has no consistent relationship with zero-shot performance.Table 8 identifies the best-performing set size in each row, but reports no general trend.Source: Table 8 · p. 16

G. Noisy Available Hierarchy Details

  • For BREEDS, CHiLS can use all ImageNet leaf labels descended from each superclass rather than only the restricted BREEDS subclasses.Some superclasses consequently receive more subclass labels than are present in the data.Source: G. Noisy Available Hierarchy Details · p. 16

H. Dataset Details

The dataset details describe domain-averaged results for BREEDS, Office31, and OfficeHome, and a manual relabeling of Fruits-360 to address uninformative names and mixed granularity.

H. Dataset Details

  • Results for BREEDS, Office31, and OfficeHome are averaged across domains, whose identities are listed in Table 10.The passage does not enumerate the domains; Table 10 is identified as their listing.Source: Table 10 · p. 16, H. Dataset Details · p. 17
  • Because Fruits-360 has classes distinguished only by numeric indices and mixed-granularity class labels, its classes were manually renamed, yielding a 59-way superclass classification with 102 ground-truth subclasses.Source: H. Dataset Details · p. 17
  • The Fruits-360 relabeling yields a 59-way superclass task with 102 ground-truth subclasses.Source: H. Dataset Details · p. 17

H.1. ObjectNet: A Case Study

The ObjectNet case study forms an 11-way coarse-grained task from 113 classes overlapping ImageNet’s hierarchy and evaluates GPT-generated subsets at m = 10. GPT-3 often misses the true subclasses, which the authors suggest may explain CHiLS’s relatively poor performance with generated subsets.

H.1. ObjectNet: A Case Study

  • ObjectNet’s coarse-grained evaluation is an 11-way task drawn from its 113 classes that overlap the ImageNet hierarchy, with GPT-generated subsets at m = 10.The task uses the BREEDS hierarchy, and the original ObjectNet classes are treated as true subclasses.Source: §H.1 · p. 17, Table 12 · p. 18
  • GPT-3 misses many true ObjectNet subclasses, including every subclass in the “equipment” and “cooked food” examples.The authors posit that this mismatch contributes to CHiLS’s relatively poor performance with GPT-generated subsets.Source: §H.1 · p. 17
  • The authors also identify ambiguous superclass names such as “accessory,” “appliance,” and “equipment,” alongside ObjectNet’s inherent difficulty, as likely contributors to poor baseline performance.Source: §H.1 · p. 17
Loading 2302.02551v3…