Source-linked AI summary
Order-Free RNN with Visual Attention for Multi-Label Classification
Shang-Fu Chen, Yi-Chen Chen, Chih-Kuan Yeh, Yu-Chiang Frank Wang
TL;DR
Existing multi-label RNN approaches require predetermined label sequences, can struggle with small objects, and may propagate prediction errors across recurrent steps. The paper jointly learns visual attention and LSTM-based label dependencies without fixed ordering, reporting favorable performance while attending to relevant regions.
Problem
Existing multi-label frameworks require predefined label orders, struggle with small objects, and can propagate recurrent prediction errors during testing.
Method
The paper jointly learns visual attention and LSTM models so label order and label-associated image regions are learned without prior ordering assumptions.
Results
The model performs favorably against state-of-the-art deep learning approaches, including when ground-truth labels are not correctly presented during training.
Takeaways & Limitations
The unified attention-LSTM architecture supports order-free label dependency learning while attending to image regions associated with each label.
Takeaways & Limitations
Prior recurrent frameworks are constrained by predefined label orders and can be sensitive to recurrent prediction errors during testing.
Abstract
from arXiv · showhide
In this paper, we propose the joint learning attention and recurrent neural network (RNN) models for multi-label classification. While approaches based on the use of either model exist (e.g., for the task of image captioning), training such existing network architectures typically require pre-defined label sequences. For multi-label classification, it would be desirable to have a robust inference process, so that the prediction error would not propagate and thus affect the performance. Our proposed model uniquely integrates attention and Long Short Term Memory (LSTM) models, which not only addresses the above problem but also allows one to identify visual objects of interests with varying sizes without the prior knowledge of particular label ordering. More importantly, label co-occurrence information can be jointly exploited by our LSTM model. Finally, by advancing the technique of beam search, prediction of multiple labels can be efficiently achieved by our proposed network model.
Introduction
Multi-label classification must model label co-occurrence while handling arbitrary label order, small objects, and recurrent prediction errors. The proposed joint attention-LSTM framework learns label order and attends to relevant image regions without requiring predetermined sequences.
- CNN-based multi-label methods remain challenging because they must extend image classification to predict multiple labels and their relationships.
- Predefined label orders may not reflect semantic dependency, making the learned sequence sensitive to how labels are ordered during training.The paper contrasts frequency-based ordering with semantically more appropriate alternatives.
- Attention is needed because objects at smaller image scales are harder to recover, but fixed recurrent ordering can attend to the wrong object first.
- Prediction errors during testing can propagate through recurrent steps because training uses ground-truth labels whereas testing uses predicted labels.
- The proposed joint attention-LSTM model learns label order automatically, attends to label-associated regions, and applies the same architecture during training and testing.Its reported advantages include attention to small objects and favorable performance against state-of-the-art deep learning approaches.
Related Work
Prior multi-label methods range from independent classifiers and dependency models to CNN and RNN architectures. RNN-based approaches can exploit label correlations, but representative CNN-RNN methods require predefined label orders and may struggle with small objects.
- Binary decomposition is simple but cannot identify relationships between labels, motivating methods that explicitly model label dependency.
- Classifier chains capture conditional label dependencies but incur high computation cost and have limited ability to model high-order correlations.
- Neural multi-label methods exploit label dependency through specialized architectures or losses, including rank-based losses, grid candidates, and label graphs.
- RNNs learn sequential connections and internal states, while LSTM variants exploit long-term label dependency for sequential multi-label prediction.
- CNN-RNN combines CNN features with LSTM inference but requires a predefined label order and may poorly recognize labels for small objects.
Our Proposed Method
The task is to predict a multi-label vector for each test input from training instances paired with binary label vectors. Each label dimension indicates whether an instance belongs to that label.
- The training data consists of N instances in a d-dimensional space paired with a multi-label matrix containing C labels.The data is represented as D = {(x_i, y_i)} from matrices X and Y.
- Each label dimension is binary, indicating whether an instance belongs to the corresponding label.
- The prediction goal is to infer the multi-label vector for a test input.
A Brief Review of CNN-RNN
CNN-RNN combines CNN feature mapping with an LSTM inference layer to model label correlations for multi-label classification. However, it requires a predefined label order and does not properly use visual attention for smaller objects.
- CNN-RNN combines a CNN feature mapping layer with an LSTM inference layer to recover correlations between labels.
- CNN-RNN requires a predefined label order during training, limiting its flexibility in learning optimal label sequences.
- Without properly utilized visual attention information, CNN-RNN may struggle to predict labels for smaller objects.
Order-Free RNN with Visual Attention
The proposed architecture jointly uses feature mapping, visual attention, and LSTM inference. Its recurrent state guides attention across regions and label steps, enabling label-correlation modeling without a fixed label order.
- The model has feature mapping, attention, and LSTM inference components for multi-label classification.Feature mapping extracts CNN visual features, attention represents image semantics, and LSTM performs inference.
- The previous LSTM hidden state updates attention to guide the network toward the next relevant image region.
- The design jointly exploits label correlations and associated visual information to automatically observe an optimal label-sequence order.
Feature Mapping Layer Mmap
The feature mapping layer uses pretrained CNNs to extract visual features and adds label-probability outputs as an auxiliary feature vector. During training, only the added fully connected layer is updated using cross-entropy while the pretrained CNN remains fixed.
- The feature mapping layer extracts visual features with pretrained CNN models and produces per-label probabilities through an added fully connected layer.The probability outputs serve as preliminary label predictions.
- The added fully connected layer is trained with log-likelihood cross-entropy while the pretrained CNN parameters remain fixed.
- Concatenated feature maps are converted into a single input vector for learning visual attention.
Attention Layer Matt
The attention layer focuses prediction on relevant image regions, especially when objects occupy small areas. It uses the previous hidden state to weight visual features and produce a context vector for recurrent inference.
- Small objects can be difficult to identify because they occupy limited or peripheral regions of an image.The paper contrasts large person instances with smaller birds near image corners.
- The attention layer focuses on appropriate image regions while predicting associated labels.It adapts a soft attention component originally used for image caption generation.
- Conditioning attention on the previous hidden state makes its feature weighting depend on recurrent inference history.
- The attention weights α_i lie in [0, 1] and represent the importance of feature i while predicting the current label.
Confidence-Ranked LSTM Mpred
The confidence-ranked LSTM combines prior confidence, attention context, and previous hard labels to model label dependencies without predefining label order. It produces sequential confidence vectors and labels for multi-label prediction.
- Confidence-Ranked LSTM Mpred: The LSTM derives each hidden state from the previous confidence vector, attention context, and previous hard label vector.This structure is intended to observe long-term dependencies between labels and exploit label correlations.
- Confidence-Ranked LSTM Mpred: The prediction layer converts the hidden state into a soft confidence vector, whose highest-confidence class becomes the hard label at each time step.Collected hard labels form the final predicted multi-label vector during testing.
- Confidence-Ranked LSTM Mpred: The output layer uses cross-entropy loss at each time step, with sigmoid-transformed predictions compared against the multi-label targets.The passages define y_i as a target label component, p_t,i as a prediction component, and σ as the sigmoid function.
- Confidence-Ranked LSTM Mpred: Applying LSTM to multi-label classification is difficult because training requires an order for the ground-truth labels.The desired order should reflect semantic dependencies among labels in the training images.
- Confidence-Ranked LSTM Mpred: The model updates its hidden state and attention layer from previous predictions, enabling attention to semantic objects without a predefined label order.The architecture uses this joint update to identify regions associated with labels.
Order-Free Training and Testing
The model performs order-free inference by selecting labels from a shrinking candidate pool and using beam search to retain multiple high-probability paths. Prediction stops when a path is too unlikely or reaches the permitted maximum length.
- Order-Free Training and Testing: The network produces a sequence of highest-confidence labels at successive time steps, supporting order-free label prediction.The candidate-pool procedure prevents repeated labels across time steps.
- Order-Free Training and Testing: At each time step, the model selects the most confident label from the candidate pool and removes it to avoid duplicate outputs.The initial pool contains all labels, and its cardinality decreases by one after each selection.
- Order-Free Training and Testing: Beam search keeps the best-K prediction paths and expands them through K × c successor paths at the next time step.This is introduced to make prediction more robust to intermediate errors that could otherwise propagate through the recurrent process.
- Order-Free Training and Testing: Each prediction path is assigned a probability based on the probabilities of its successive predicted labels.The path probability is evaluated for candidate sequences during beam-search inference.
- Order-Free Training and Testing: Beam-search prediction terminates when a path probability falls below a cross-validated threshold or its length reaches the largest training-set label count.These conditions bound inference by confidence and by a predefined maximum path length.
- Order-Free Training and Testing: Table 1 evaluates NUS-WIDE using macro and micro precision, recall, and F1 scores across the proposed and controlled variants.The variants include removing attention and using frequency-first or rare-first predefined label orders.
Experiments
Experiments on NUS-WIDE and MS-COCO compare the proposed attention-based model with baselines and controlled variants. Results favor joint visual attention and learned label ordering, including under noisy annotations.
- NUS-WIDE: The full model performed favorably against WARP, CNN-RNN, and controlled variants on NUS-WIDE.The comparisons include removing attention and fixing label order by frequency-first or rare-first strategies.
- Qualitative analysis: Attention maps identified attended regions while predictions remained reasonable for some labels judged incorrect because of noisy ground truth.The examples include both correct and incorrect predictions on NUS-WIDE and MS-COCO.
- Qualitative analysis: The model predicted visually salient animals and horses before the frequent label person, unlike CNN-RNN's frequency-defined ordering.
- MS-COCO: 4.1% in C-F1 and 5.6% in O-F1 were the improvements over the ResNet-based baseline on MS-COCO.
- Qualitative analysis: Hidden states combined with attention enabled correct surfboard prediction even when the attention map included surrounding regions.This illustrates use of both local and global image information during multi-label prediction.
Conclusion
The paper proposes a visual attention model with a confidence-ranked LSTM for multi-label classification. Joint learning supports learned label sequences and robustness to noisy annotations.
- The proposed model combines visual attention with a confidence-ranked LSTM for multi-label classification.
- Joint learning allows proper label sequences with visually attended regions without predetermined label orders.
- Experiments support effectiveness and robustness when training data are noisy or incorrectly annotated.