Source-linked AI summary
General Multi-label Image Classification with Transformers
Jack Lanchantin, Tianlu Wang, Vicente Ordonez, Yanjun Qi
TL;DR
Multi-label image classification requires predicting structured sets of labels while handling dependencies and incomplete observations. C-Tran uses a Transformer with ternary label-state masking, achieves state-of-the-art results across its evaluated settings, and supports partial or extra labels at inference.
Problem
Multi-label image classification must predict sets of labels while exploiting dependencies among visual features and labels.
Method
C-Tran trains a Transformer encoder with label mask training to predict target labels from visual features and masked labels encoded as positive, negative, or unknown.
Results
C-Tran outperforms state-of-the-art methods across regular, partially observed, and extra-label inference scenarios.
Takeaways & Limitations
The unified model supports arbitrary amounts of partial or extra labels during inference and uses interventions as partial evidence for interpretable, accurate predictions.
Takeaways & Limitations
Transformer layers are memory-intensive, limiting the model to around 2000 labels with four NVIDIA Titan X GPUs; training assumes no more than 75% of labels are known.
Abstract
from arXiv · showhide
Multi-label image classification is the task of predicting a set of labels corresponding to objects, attributes or other entities present in an image. In this work we propose the Classification Transformer (C-Tran), a general framework for multi-label image classification that leverages Transformers to exploit the complex dependencies among visual features and labels. Our approach consists of a Transformer encoder trained to predict a set of target labels given an input set of masked labels, and visual features from a convolutional neural network. A key ingredient of our method is a label mask training objective that uses a ternary encoding scheme to represent the state of the labels as positive, negative, or unknown during training. Our model shows state-of-the-art performance on challenging datasets such as COCO and Visual Genome. Moreover, because our model explicitly represents the uncertainty of labels during training, it is more general by allowing us to produce improved results for images with partial or extra label annotations during inference. We demonstrate this additional capability in the COCO, Visual Genome, News500, and CUB image datasets.
1. Introduction
C-Tran uses a Transformer encoder and ternary label-state masking to model dependencies among visual features and labels. The framework supports standard prediction as well as inference conditioned on partial or extra labels.
- Multi-label classification must model dependencies among visual features and labels because images contain multiple objects and complex situations.
- C-Tran reconstructs target labels from convolutional visual features and randomly masked label embeddings using a Transformer encoder.
- The label mask objective represents each label as positive, negative, or unknown during training.
- At inference, C-Tran can use only visual features or combine them with partial target labels or related extra labels.
- The framework is evaluated across six datasets and three inference settings, achieving state-of-the-art results across all six datasets.
- State embeddings support user interventions as partial evidence, enabling more interpretable and accurate predictions.
2. Problem Setup
The problem setup distinguishes regular prediction from inference with partial or extra labels. C-Tran masks labels during training, encodes known labels by state, and predicts the unknown labels.
- Regular Multi-label Classification: Regular multi-label classification predicts the complete set of binary labels from an input image.
- Inference with Partial Labels: With partial labels, the model predicts unknown target labels from image features and an observed subset of target labels.
- Inference with Extra Labels: With extra labels, related non-target information is appended to the target label set and can affect predictions of unknown target labels.
- Label-State Encoding: C-Tran represents label states as unknown, negative, or positive using learned state embeddings, with the unknown vector fixed to zeros.
3. Method: C-Tran
C-Tran jointly models image features and label embeddings with a Transformer, while state embeddings encode labels as unknown, negative, or positive. Label Mask Training exposes the model to varied known-label combinations so it can support regular, partial-label, and extra-label inference.
- C-Tran predicts unknown labels from image features and known-label information across regular, partial-label, and extra-label inference settings.Regular inference supplies no known labels; partial-label inference supplies known target labels; extra-label inference supplies labels outside the target set.
- C-Tran feeds image-region, label, and state-enhanced embeddings jointly into a Transformer encoder to model dependencies among visual features and labels.Self-attention computes pairwise importance weights, updates embeddings using weighted sums and a nonlinear layer, and repeats this process across encoder layers.
- The model represents each possible label with a learned label embedding and adds a learned state embedding indicating unknown, negative, or positive status.The unknown state vector is fixed to all zeros, while positive and negative states encode known label values.
- Independent feedforward classifiers transform the final label representations into the model’s label predictions.Each label uses a separate single-layer feedforward network with a label-specific weight vector and sigmoid output.
- Label Mask Training randomly masks between 0.25ℓ and ℓ labels, using ground-truth positive or negative states for the remaining labels to reconstruct the masked labels.The objective approximately minimizes expected cross-entropy over known-label configurations, teaching the model varied combinations of known and unknown labels.
- Masking at least 0.25ℓ labels targets inference with zero to 0.75ℓ known labels, because the authors assume knowing more than 75% is unrealistic.This training range is intended to let C-Tran incorporate arbitrary amounts of known information during inference within that stated scope.
4. Experimental Setup and Results
The experiments evaluate C-Tran across regular, partial-label, and extra-label settings, with results showing strong performance and benefits from label mask training. The analysis also identifies a computational boundary caused by memory-intensive Transformer layers.
- Regular inference: C-Tran achieves state-of-the-art performance across almost all metrics on COCO-80 and VG-500.Absolute mAP increases of 0.8 and 1.0 are reported for COCO-80 and VG-500, respectively.
- Inference with partial labels: C-Tran outperforms Feedbackprop at every tested amount of partial labels across all four datasets.Its improvement over Feedbackprop increases as the percentage of known partial labels rises.
- Inference with extra labels: C-Tran consistently achieves the best CUB-312 performance when any amount of extra label groups is available.The multi-task baseline performs best with no known groups but cannot incorporate known extra labels, whereas C-Tran can.
- Ablation and model analysis: Without image features, C-Tran still learns rich dependencies from label annotations.The no-image ablation obtains relatively high mAP on NEWS-500 and COCO-1000.
- Ablation and model analysis: Label mask training improves performance in both regular and 50% partial-label inference settings.It also produces more meaningful semantic topology in learned label embeddings, clustering objects from the same group together.
- Ablation and model analysis: Transformer depth is relatively robust at 2, 3, and 4 layers, with COCO-80 mAP scores of 85.0, 85.1, and 84.3, respectively.Transformer layers are memory-intensive; on four NVIDIA Titan X GPUs, the model supports around 2000 labels.
5. Related Work
The related work organizes multi-label classification around conditional prediction, shared embedding spaces, and structured outputs, while distinguishing C-Tran's treatment of partial and extra labels. C-Tran represents label states as positive, negative, or unknown to support arbitrary known-label sets during inference.
- Multi-label image classification: Multi-label classification predicts sets of objects, attributes, or actions, whose structure reflects relationships among concepts in real-world images.It differs from single-label multi-class classification, which associates each image with one concept.
- Multi-label image classification: Existing multi-label methods include conditional prediction, shared embedding spaces, and structured-output approaches.Conditional prediction models estimate joint label probabilities sequentially, while shared-embedding methods align image features and labels.
- Inference with partial labels: Feedbackprop handles arbitrary partial labels at test time but requires iterative inference and is not trained on partial evidence.Other methods using partial labels rely on a fixed set of labels and therefore do not generalize to arbitrary known-label sets.
- Inference with partial labels: C-Tran encodes each label as positive, negative, or unknown, enabling arbitrary amounts of partial evidence during both training and inference.This state representation is presented as the first learning method supporting arbitrary partial evidence in both phases.
- Inference with extra labels: Concept Bottleneck Models use fixed intermediate concepts and assume those concepts are sufficient features for target prediction.C-Tran instead uses state embeddings for known or unknown concepts without assuming how many concepts will be known at inference.
6. Conclusion
C-Tran is presented as a flexible, resource-efficient framework for multi-label image classification across regular, partially observed, and extra-label settings. The authors report state-of-the-art performance in all evaluated scenarios and attribute gains to modeling label and image-feature interactions.
- C-Tran supports regular multi-label classification as well as inference with partially observed or extra labels.The same framework is intended for varied multi-label image-classification applications.
- C-Tran requires no extra resources and is described as easy to implement.
- C-Tran outperforms all state-of-the-art methods in all reported scenarios.
- The analysis attributes performance improvements to explicitly modeling interactions between target labels and between image features and target labels.
- The authors identify hierarchical scene categorization and training for labels never observed during training as future directions.
A.1. Qualitative Examples
Qualitative examples illustrate how C-Tran uses partial or extra labels known before inference to alter its predictions on COCO-80 and CUB-312.
- On COCO-80, Figure 5 compares ResNet-101, C-Tran without partial labels, and C-Tran given ϵ = 25% partial labels.The observed partial-label setting supplies 21 known labels before inference.
- Some COCO-80 labels cannot be predicted well without partial labels, according to the qualitative examples.
- On CUB-312, Figure 6 compares C-Tran without extra labels with predictions using ϵ = 54% of extra labels.The extra labels are high-level bird-species concepts rather than target labels, and the setting supplies 60 labels before inference.
A.2. Detailed Diagram of C-Tran Settings
Figure 7 diagrams the paper’s possible training and inference settings and shows how C-Tran applies across them using the same random-mask training.
- Figure 7 presents a detailed diagram of all possible training and inference settings used in the paper.
- The same random-mask training allows C-Tran to be applied to any of the three inference settings.
A.3. Multi-Label Classification Metrics
The metric notation defines label counts used to evaluate multi-label classification, including true positives, predicted images, and ground-truth images for each label.
- C denotes the number of labels in the metric notation.
- N_i^c denotes the true positives for the i-th label.
- N_i^p denotes the total number of images for which the i-th label is predicted.
- N_i^g denotes the number of ground-truth images for the i-th label.
A.4. More Discussions of C-Tran
C-Tran connects Transformer self-attention with joint image-feature and label modeling, while supporting partial evidence during inference. Its fully connected attention and graph-attention views distinguish it from prior graph-based and encoder/decoder approaches.
- Connecting to Transformers and BERT: C-Tran applies self-attention to feature and label embeddings rather than only sentence word embeddings.This adapts Transformer mechanisms from language processing to multi-label image classification.
- Connecting to Transformers and BERT: Unlike encoder/decoder object-detection Transformers, C-Tran jointly models image and label features in one Transformer encoder.The image features can be updated conditioned on the labels.
- Connecting to Transformers and BERT: C-Tran uses unique state embeddings to incorporate partial evidence during image classification.This capability is identified as a distinction from prior computer-vision Transformer approaches.
- Qualitative examples: Figures 5 and 6 provide qualitative examples with partial labels on COCO-80 and extra labels on CUB-312.Figure 5 uses ϵ = 25% partial labels, while Figure 6 uses ϵ = 54% extra labels.
- Connecting to Graph Based Neural Relational Learning: C-Tran uses graph attentions so each target label can attend differentially to relevant parts of the input image.The approach addresses limitations of methods that require expensive bounding-box annotations or obtain only rough regions of interest.
- Connecting to Graph Based Neural Relational Learning: C-Tran’s Transformer encoder forms a fully connected graph that can learn relationships between visual features and labels.This contrasts with prior graph methods that do not incorporate partial evidence into prediction.