PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-task CNN Model for Attribute Prediction

Abrar H. Abdulnabi, Gang Wang, Jiwen Lu, Kui Jia

arXiv:1601.00400v1cs.CV

TL;DR

Semantic attribute prediction is challenging because existing independent classifiers often neglect relationships among co-occurring attributes and multi-label combinations grow rapidly. The paper addresses this with multi-task CNNs, a shared latent task matrix, parameter decomposition, and group-aware sharing and competition. Experiments on AwA and Clothing Attributes report effectiveness over standard methods, with AwA performance exceeding approximately 75% prior state-of-the-art results by a large margin.

  • Problem

    Existing attribute-prediction literature mostly uses independent classifiers and rarely models co-occurrence relationships, while M attributes create 2^M possible label combinations.

  • Method

    The method trains multi-task CNNs that share deep visual knowledge through a latent task matrix and combination matrix, with grouping encouraging within-group sharing and between-group competition.

  • Results

    The method is effective on the AwA and Clothing Attributes datasets and exceeds approximately 75% previous state-of-the-art performance on AwA by a large margin.

  • Takeaways & Limitations

    Shared latent features allow attribute classifiers to leverage visual knowledge from other tasks, including under-sampled classifiers, while group information guides selective sharing.

Abstract

from arXiv · show

This paper proposes a joint multi-task learning algorithm to better predict attributes in images using deep convolutional neural networks (CNN). We consider learning binary semantic attributes through a multi-task CNN model, where each CNN will predict one binary attribute. The multi-task learning allows CNN models to simultaneously share visual knowledge among different attribute categories. Each CNN will generate attribute-specific feature representations, and then we apply multi-task learning on the features to predict their attributes. In our multi-task framework, we propose a method to decompose the overall model's parameters into a latent task matrix and combination matrix. Furthermore, under-sampled classifiers can leverage shared statistics from other classifiers to improve their performance. Natural grouping of attributes is applied such that attributes in the same group are encouraged to share more knowledge. Meanwhile, attributes in different groups will generally compete with each other, and consequently share less knowledge. We show the effectiveness of our method on two popular attribute datasets.

I. INTRODUCTION

Semantic attributes provide detailed object knowledge and support applications such as knowledge transfer, search, and recommendation, but independent attribute classifiers often overlook relationships among co-occurring attributes. The paper proposes an enhanced multi-task CNN framework that shares visual knowledge, incorporates attribute grouping, and decomposes parameters into latent and combination matrices.

  • Motivation: Semantic attributes describe objects in detail and bridge low-level visual features with high-level categories.They support knowledge transfer, information sharing, content analysis, recommendation, search, and retrieval.
  • Motivation: Most existing methods independently train discriminative classifiers, while relatively few model relationships among attributes that may co-occur.
  • Motivation: Multi-label attribute prediction is difficult because M attributes yield 2^M possible label combinations.
  • Proposed approach: The proposed framework uses deep CNN features and multi-task learning so attribute models can share visual knowledge.For many attributes, the framework applies multi-task learning to features generated by separately trained CNNs.
  • Evaluation: Experiments on the AwA and Clothing Attributes datasets demonstrate the method’s effectiveness compared with standard methods.The Clothing dataset is trained with simultaneous multi-task CNNs, while AwA uses separately trained CNN features followed by the multi-task framework.
  • Proposed approach: Attributes in the same group are encouraged to share feature statistics, whereas attributes in different groups are discouraged from sharing knowledge.The model decomposes parameters into a latent task matrix and a linear combination weight matrix without requiring mutually exclusive groups.

II. RELATED WORK

Related work defines semantic, global, localized, and correlated attributes and discusses multi-label learning as assigning multiple labels to an image. Attribute correlation motivates grouping classifiers that share feature structure.

  • Semantic Attributes: A semantic attribute is a visual property that appears or disappears and can be expressed in human language.
  • Attribute Types: Global attributes describe holistic image properties, whereas localized attributes describe specific object parts or locations.
  • Attribute Correlations: Correlated attributes co-occur or imply one another and may occupy nearby regions of feature space, making natural grouping useful.
  • Multi-labeling: Image multi-labeling assigns multiple labels to one image, but increasing labels make possible combinations intractable.A common transformation splits the problem into multiple single binary classifiers.

B. Multi-task learning

Prior multi-task learning methods share knowledge across correlated or under-sampled tasks, while CNNs learn robust visual features but are costly to train for multi-label prediction. The paper combines multi-task CNNs with deep features and a sharable latent task matrix.

  • Multi-task Learning: Multi-task learning imposes knowledge sharing while solving multiple correlated tasks simultaneously, potentially improving some or all tasks.
  • Task and Feature Correlations: Existing approaches seek task relationships, common feature structures, or both through methods including max-margin and Bayesian frameworks.
  • CNN Feature Learning: CNNs learn high-level visual abstractions through multiple nonlinear transformations but require substantial training data and time.
  • CNN and Multi-labeling: Direct CNN training for multi-label prediction is described as infeasible or impractical, motivating multi-task CNN alternatives.
  • Proposed Direction: The proposed approach trains multi-task classifiers on deep features and uses a sharable latent task matrix for attribute prediction.The latent matrix is presented as potentially informative for generating a full attribute description of an image.

III. MULTI-TASK CNN MODELS

The model assigns one binary attribute to each CNN, feeds attribute-specific convolutional features into a joint multi-task loss, and decomposes classifier weights into shared latent and CNN-specific combination components. This enables shared visual patterns while retaining task-specific prediction weights.

  • Model Structure: Each of M CNN models predicts one binary, nameable attribute from an input image.
  • Model Structure: Features from the last convolution layers are fed into a joint multi-task loss layer for attribute prediction.
  • Latent Task Decomposition: The classifier weight matrix W is decomposed as W = LS, where L is shared and each column of S corresponds to one CNN classification layer.
  • Latent Task Decomposition: The latent matrix lets CNNs share visual patterns and collaborate during training, helping classifiers with insufficient samples leverage knowledge from other tasks.

B. Feature Sharing and Competition in MTL

The framework uses attribute groups as side information to promote feature sharing within groups and competition across groups. Its regularization and latent decomposition also accommodate overlapping groups and localized features.

  • Group membership encourages classifiers for same-group attributes to share visual knowledge, while different groups tend to compete and share less.
  • The framework imposes no mutual-exclusion restriction, allowing groups to overlap while latent configurations mitigate overlap through localized features.
  • L1 and L21 regularization balance task competition and feature sharing by promoting sparsity and shared feature-dimension selection.
  • An adapted L21 term applies sharing within groups and competition between groups over latent task dimensions.
  • The vector s_g^k collects attribute vectors within a group, encouraging same-group classifiers to share selected latent dimensions while different groups compete.

C. Formulations of the Multi-task CNN model

The model decomposes overall parameters into a latent task matrix and combination matrix, optimizing attribute-specific losses with structured regularization. CNN-generated features feed classifiers that predict binary attributes.

  • The objective combines squared hinge loss with group-aware regularization over latent tasks and combination weights.
  • For attribute m, training uses N_m feature-label samples with labels in {-1,+1}, while K denotes the latent task dimension.
  • Group regularization encourages intra-group sharing, inter-group competition, latent-matrix sparsity, and protection against overfitting.
  • At testing, each image is processed by all CNNs, and corresponding classifier weight vectors produce attribute predictions.
  • When attributes are numerous, bottom CNN layers are frozen while the multi-task loss predicts attributes from generated features.

D. Optimization Steps

Optimization alternates between the combination matrix and latent task matrix because the joint objective is non-convex but convex in either block when the other is fixed. The resulting weights are reconstructed for CNN training.

  • Shared multi-task training: Each CNN predicts one attribute, while a shared layer and optimized components of W enable visual knowledge sharing across classifiers.
  • Alternating optimization: The non-convex objective is optimized alternately: APG updates L when S is fixed, and SPGD updates S when L is fixed.
  • Algorithm 1: Algorithm 1 fixes L to optimize S, fixes S to optimize L, and repeats these steps until convergence.
  • Alternating optimization: SPGD smooths the mixed-norm objective; squaring the mixed-norm term preserves the solution path while making optimization easier.
  • Optimization properties: APG handles non-smooth convex functions through proximal shrinkage, whereas its convergence rate is described as relatively lower.
  • CNN integration: After optimizing the multi-task objective, W is reconstructed from L and S and returned with gradients for the backward pass.

A. Datasets and Grouping

Experiments use the Clothing Attributes Dataset and Animals with Attributes, which differ substantially in attribute count and grouping structure. The datasets provide binary attribute annotations for image or class-level prediction.

  • Clothing Attributes Dataset: The Clothing Attributes Dataset contains 1,856 images, 23 binary attributes, and three multiclass attributes that are excluded.
  • Clothing Attributes Dataset: Clothing annotations are provided at image level, with every image annotated against all attributes.
  • Animals with Attributes: The AwA dataset contains 30,475 images from 50 animal classes and provides 85 binary attributes per class.
  • Animals with Attributes: AwA annotations are class-level and its attributes are organized into nine groups, including colors, textures, shapes, behavior, habitat, and body parts.

B. Attribute Prediction Accuracy

The multi-task framework improves attribute prediction on Clothing and AwA, outperforming standard and single-task methods, with especially large gains on AwA.

  • Clothing Attributes Dataset: MG-CNN outperforms state-of-the-art results on the Clothing Attributes Dataset.Table III compares pre-sharing, post-sharing, and previous methods across color, pattern, cloth-parts, and appearance groups.
  • Clothing Attributes Dataset: The Clothing experiment compares feature-extraction, single-task CNN, ungrouped multi-task, grouped multi-task, and prior baselines.S-extract and M-extract use pre-trained CNN features, while S-CNN, M-CNN, and MG-CNN differ in task sharing and group encoding.
  • AwA: AwA evaluation reports mean average precision over all attributes and compares the multi-task framework with standard methods.Table IV presents attribute detection scores, with higher mean average precision indicating better performance.
  • AwA: The AwA multi-task CNN models outperform single-task models by a large margin and exceed prior state-of-the-art results of approximately 75%.The AwA experiment also reports mean average precision for individual attribute groups before and after applying the multi-task framework.
  • Qualitative examples: Multi-task classifiers correctly classify examples that single-task classifiers misclassify in both AwA and Clothing datasets.Figure 3 includes four AwA rows and two Clothing rows, with Yes/No indicating attribute presence or absence.

C. Implementation Details

The experiments initialize CNNs from ImageNet, train them for up to 100 epochs, and optimize latent-task sharing with specified regularization and validation settings.

  • CNN model training: CNN models are initialized from an ImageNet-pretrained network and fine-tuned on target attribute annotations for 100 epochs.Most models converged in approximately 50 epochs; inputs are resized to 256x256 and mean-normalized.
  • CNN model training: Clothing training uses data augmentation following the cited CNN training procedure.The augmentation strategy is applied specifically when training on the Clothing dataset.
  • Multi-task optimization: The multi-task optimization initializes L with SVD of W, randomly initializes S, and constructs W by stacking final fully connected layers.Other parameters are selected experimentally or with standard heuristics.
  • Multi-task optimization: The latent-task dimension is set to 2048, the maximum possible feature dimension, to avoid severe information loss from SVD.The choice reflects the importance of preserving subtle fine-grained details in attribute prediction.
  • Optimization settings: CNN training uses weight decay 0.0005, momentum 0.9, and an initially 0.01 learning rate that is manually reduced.The latent-task regularization parameter λ is 0.4, while γ and µ are validated separately for each dataset.
  • Computational cost: On two NVIDIA TK40 16GB GPUs, Clothing training takes approximately 1.5 days and sequential testing takes approximately 50 minutes.Feature extraction takes about 1.5 minutes per 1000 images per model.

V. CONCLUSION

The paper introduces an enhanced multi-task CNN framework for binary semantic attribute prediction. It shares visual knowledge, encodes semantic groups, and decomposes parameters into latent task and combination matrices.

  • Contribution: The method jointly predicts binary semantic attributes while sharing visual knowledge between tasks.Each task uses attribute-specific representations within the multi-task framework.
  • Contribution: Semantic group information encourages greater sharing within groups while supporting competition between different groups.The framework uses natural attribute groupings to structure knowledge sharing.
  • Contribution: Model parameters are decomposed into a latent task matrix and a linear combination matrix.The latent task matrix learns localized feature patterns that can be shared across classifiers.
  • Implication: Under-sampled classifiers can generalize better by leveraging the sharable latent layer.The paper identifies the latent task matrix as a topic for future investigation, including semantic or latent image descriptions.
  • Results: Experiments on two benchmark attribute datasets show that the multi-task CNN classifiers outperform previous single-task classifiers.The conclusion reports this outcome across both evaluated datasets.
Loading 1601.00400v1…