PaperlayerView exact revision on arXiv

Source-linked AI summary

Orderless Recurrent Models for Multi-label Classification

Vacit Oguz Yazici, Abel Gonzalez-Garcia, Arnau Ramisa, Bartlomiej Twardowski, Joost van de Weijer

arXiv:1911.09996v3cs.CV

TL;DR

Multi-label recurrent models must output labels sequentially, although the natural label order can vary by image and fixed orders can penalize correct predictions. This paper dynamically aligns ground-truth labels with predicted sequences, and reports faster training, fewer duplicate predictions, and state-of-the-art results on several datasets.

  • Problem

    Recurrent multi-label models are penalized when correct labels are predicted in an order different from the ground-truth sequence, complicating convergence and training.

    Source: §1 · p. 2

  • Method

    PLA and MLA dynamically reorder ground-truth labels to align them with the LSTM's predicted sequence during training.

    Source: §1 · p. 2, §3.3 · pp. 4–5

  • Results

    The orderless approach trains faster, seldom generates duplicate predictions, and achieves state-of-the-art multi-label results on several datasets.

    Source: §5 · p. 8

  • Takeaways & Limitations

    A standard CNN-RNN architecture combined with an orderless loss achieves state-of-the-art results for multi-label classification on several datasets.

    Source: §5 · p. 8

    MLA can assign a label to a step where it is already predicted, producing gradients that favor it over another ground-truth label.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Recurrent neural networks (RNN) are popular for many computer vision tasks, including multi-label classification. Since RNNs produce sequential outputs, labels need to be ordered for the multi-label classification task. Current approaches sort labels according to their frequency, typically ordering them in either rare-first or frequent-first. These imposed orderings do not take into account that the natural order to generate the labels can change for each image, e.g.\ first the dominant object before summing up the smaller objects in the image. Therefore, in this paper, we propose ways to dynamically order the ground truth labels with the predicted label sequence. This allows for the faster training of more optimal LSTM models for multi-label classification. Analysis evidences that our method does not suffer from duplicate generation, something which is common for other models. Furthermore, it outperforms other CNN-RNN models, and we show that a standard architecture of an image encoder and language decoder trained with our proposed loss obtains the state-of-the-art results on the challenging MS-COCO, WIDER Attribute and PA-100K and competitive results on NUS-WIDE.

1. Introduction

Multi-label classification must model dependencies among visual concepts, but recurrent models penalize correct labels when their sequence differs from the ground-truth order. The paper dynamically aligns ground-truth labels with predictions, reporting faster training, fewer duplicate predictions, and state-of-the-art results on several datasets.

1. Introduction

  • PLA correctly assigns labels in examples where rare-first or frequent-first ordering can cause labels to be ignored.The examples contrast a missed frisbee under rare-first ordering with a missed person under frequent-first ordering.Source: §1 · p. 1
  • Multi-label classification assigns images multiple visual concepts, whose dependencies and similarities make label prediction a structured problem.For example, a model may be sure an image contains either a bicycle or motorcycle, but not both.Source: §1 · p. 1
  • A recurrent decoder can penalize correct predictions made in an order different from the ground truth, hindering convergence and model quality.A prominent dog predicted first can be penalized if it is not first in the imposed sequence, then penalized again for missing its assigned step.Source: §1 · p. 2
  • Dynamic label alignment with PLA or MLA trains faster, avoids repeated predictions, and achieves state-of-the-art results on several datasets.The paper reports state-of-the-art results on MS-COCO, WIDER Attribute, and PA-100K.Source: §1 · p. 2

2. Related Work

Prior work uses recurrent networks to model label dependencies, but existing approaches either impose fixed label orders or do not dynamically align labels to predictions. The paper places this work alongside recurrent methods for orderless data and encoder-decoder approaches for image captioning.

2. Related Work

  • Earlier orderless recurrent work explored architectures and training procedures for sets, including recurrent methods for multi-label image annotation.The cited approaches do not establish that label order is dynamically adapted to the model's predictions.Source: §2 · pp. 2–3
  • Some earlier multi-label methods treated labels independently, while others modeled correlations using graphical models or chains of binary classifiers.Explicit pairwise correlation modeling was described as relatively computationally costly.Source: §2 · p. 2
  • RNN-based multi-label methods model label correlations without the high computation costs of explicitly modeling pairwise dependencies.Earlier work used frequent-first, rare-first, dictionary, or random orders; another approach used BCE at each time step but did not dynamically adapt the order.Source: §2 · pp. 2–3
  • CNN-RNN multi-label models encode images with a CNN and use an RNN to generate label sequences that model label dependencies.Prior work varied the visual representation and label ordering supplied to the recurrent model.Source: §2 · p. 2
  • Image-captioning systems commonly use a CNN encoder and recurrent decoder trained end-to-end to generate a sentence from image features.The architecture in this paper uses an image CNN encoder, LSTM decoder, attention, and an orderless loss.Source: §2 · p. 3, Figure 2 · p. 3

3. Method

The model encodes an image with a CNN and decodes label sequences with an RNN, while orderless training aligns ground-truth labels to predictions before computing loss.

3.1. Image-to-sequence model

  • The CNN initializes the RNN hidden state, which predicts labels sequentially until an end signal; attention combines convolutional features with the prior predicted label embedding at each step.The attention module uses fourth-layer activations and the current LSTM hidden state to focus on different image regions, addressing the difficulty of representing small objects with global features.Source: §3.1 · p. 3

3.2. Training recurrent models

  • Fixed label order can penalize correct predictions made in another order, while larger or more frequent classes can cause other labels to be ignored.The illustrated default-order losses are 7.59 for person/tie and 6.27 for umbrella/person/surfboard.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3. Orderless recurrent models

  • Training aligns ground-truth labels to network predictions before computing loss, using minimal-loss alignment or predicted-label alignment.The standard recurrent loss compares each step’s prediction with the ground-truth label assigned to that same step.Source: §3.3 · pp. 4–5, §3.2 · p. 4
  • MLA uses Hungarian assignment to minimize summed cross-entropy, while PLA fixes correct predictions and assigns the remaining labels, accepting higher loss for greater prediction consistency.In the illustrated case, MLA assigns chair to t3 despite the model predicting it at t4, which can misdirect gradients.Source: §3.3 · pp. 4–5, Figure 4 · p. 5

4. Experiments

Experiments compare dynamic label alignment with fixed-order strategies, assess order behavior and computation, and benchmark the approach on four datasets.

4.1. Datasets and setting

  • Evaluation covers MS-COCO, NUS-WIDE, WIDER Attribute and PA-100K, using per-class and overall precision, recall and F1, plus image-wise accuracy on PA-100K.Source: §4.1 · p. 5

4.2. Comparison ordering approaches and analysis

  • MLA and PLA train faster and achieve lower overall loss than rare-first and frequent-first strategies, which can penalize correct labels predicted in a different order.That order-only penalty can drive gradients toward a predefined sequence even when it is suboptimal for an image.Source: §4.2 · p. 6, Table 2 · p. 7
  • MLA and PLA produce no duplicate labels, while their 80.25% and 82.87% order-rigidness scores indicate that predicted label order varies across images.Fixed-order methods score 100.00% on order-rigidness.Source: §4.2 · p. 6, Table 1 · p. 6
  • PLA alignment is faster because it applies the Hungarian algorithm only to wrongly predicted labels; Table 3 reports per-image times for ResNet-101 on MS-COCO.Source: §4.2 · p. 7, Table 3 · p. 7

4.3. Experimental results

  • PLA ultimately achieves lower loss and better accuracy than MLA, and the attention model significantly improves results.PLA aligns labels with actual predictions; MLA aligns them to minimize loss.Source: Figure 5 · p. 6, §4.3 · p. 7
  • For MS-COCO sports, BCE has larger co-occurrence differences from ground truth, including skis/snowboards and sports ball/frisbee; PLA predictions are more realistic.Source: §4.3 · pp. 7–8, Figure 6 · p. 7
  • On MS-COCO and NUS-WIDE, the method surpasses other CNN-RNN models; MLIC-KD-WSD achieves globally best overall and per-class F1 on NUS-WIDE.The reported comparisons are in Tables 4 and 5.Source: §4.3 · p. 8, Table 4 · p. 8, Table 5 · p. 8
  • On WIDER Attribute and PA-100K, the CNN-RNN model surpasses other models despite label correlations being uncommon in these attribute datasets.The comparisons are reported in Tables 6 and 7.Source: §4.3 · p. 8, Table 6 · p. 8, Table 7 · p. 8

5. Conclusions

The paper trains orderless LSTMs by dynamically ordering labels from predictions, reporting faster loss minimization, few duplicate predictions, and state-of-the-art results on several datasets.

5. Conclusions

  • The proposed orderless loss gives a standard CNN-RNN architecture state-of-the-art multi-label classification results on several datasets.Source: §5 · p. 8
  • The two alternative losses dynamically order labels based on the LSTM's predictions rather than imposing frequent-first or rare-first ordering.Source: §5 · p. 8
  • The approach minimizes loss faster than other methods and seldom generates duplicate predictions.Source: §5 · p. 8

1. Introduction

The supplementary material presents examples comparing PLA with frequent-first label ordering and co-occurrence matrices from BCE and LSTM models on MS-COCO.

1. Introduction

  • The supplementary material compares image-label orders produced by predicted label alignment (PLA) and the frequent-first approach.Source: §1 · p. 11
  • It includes co-occurrence matrices computed from the best BCE and LSTM models.Source: §1 · p. 11
  • The matrices cover different MS-COCO super-categories.Source: §1 · p. 11

2. Qualitative comparison of PLA and other methods

The examples show that a fixed frequent-first ordering can confuse predictions when an image contains dominant but less frequent objects or minor but more frequent ones, producing duplicates.

2. Qualitative comparison of PLA and other methods

  • Frequent-first ordering predicts labels in the same order across images, even when their image-specific prominence differs.Source: §2 · p. 11
  • The fixed order can confuse predictions when an image contains a dominant but less frequent object or a minor but more frequent one.Source: §2 · p. 11
  • That confusion can lead to duplicate predictions at different time steps.Source: §2 · p. 11

3. Additional co-occurrence matrices of LSTM and BCE models

The co-occurrence comparisons report higher levels for BCE than PLA, attributed to repeated use of image evidence for similar objects; the examples also contrast their label orders and predictions.

3. Additional co-occurrence matrices of LSTM and BCE models

  • BCE has higher predicted class co-occurrence than PLA, with false positives linked to reusing image evidence for different objects.The reported confusion is especially visible in animals, food, vehicles, and kitchen categories.Source: §3 · p. 11, Figure 3 · p. 13
  • The LSTM model has smaller differences from ground truth because it takes previous predictions into account at each time step.Source: §3 · p. 11
  • The figures compare the label orders yielded by PLA and frequent-first, with wrong or duplicate predictions underlined.Source: Figure 1 · p. 11
  • In the person–skis example, frequent-first predicts skis twice, while PLA predicts person and skis once each.The frequent-first loss is 0.54 and the PLA loss is 0.23.Source: §3 · p. 12
Loading 1911.09996v3…