Source-linked AI summary
Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification
Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, Rogerio Feris
TL;DR
Deep multi-task networks need to share features among related tasks without relying on tedious, designer-biased manual architecture search. The paper dynamically widens a thin network to group tasks during training, producing compact models that preserve strong person-attribute accuracy while reducing cost.
Problem
Hand-designed multi-task architectures make selective feature sharing difficult to determine because the architecture space is combinatorially large and manual exploration is tedious.
Method
The method starts with a thin network and greedily widens it during training using task-grouping decisions that account for task relatedness and model complexity.
Results
On CelebA, the method matches 90% facial-attribute accuracy with a model 90x more compact and 3x faster than VGG-16, while preserving accuracy for facial and clothing prediction.
Takeaways & Limitations
The produced multi-task models provide compact, fast person-attribute prediction while automatically learning whom tasks should share features with.
Takeaways & Limitations
The current design cannot change the width of an individual branch, although some intrinsically difficult tasks may require wider branches.
Abstract
from arXiv · showhide
Multi-task learning aims to improve generalization performance of multiple prediction tasks by appropriately sharing relevant information across them. In the context of deep neural networks, this idea is often realized by hand-designed network architectures with layers that are shared across tasks and branches that encode task-specific features. However, the space of possible multi-task deep architectures is combinatorially large and often the final architecture is arrived at by manual exploration of this space subject to designer's bias, which can be both error-prone and tedious. In this work, we propose a principled approach for designing compact multi-task deep learning architectures. Our approach starts with a thin network and dynamically widens it in a greedy manner during training using a novel criterion that promotes grouping of similar tasks together. Our Extensive evaluation on person attributes classification tasks involving facial and clothing attributes suggests that the models produced by the proposed method are fast, compact and can closely match or exceed the state-of-the-art accuracy from strong baselines by much more expensive models.
1. Introduction
The paper proposes automated, selective feature sharing for compact multi-task deep networks, replacing tedious manual architecture exploration with dynamic branching guided by task relatedness and model complexity.
- Contribution: The approach automatically learns branching architectures and which tasks should share features.Branching and task-grouping decisions are made at each network layer.
- Design goals: The method targets low memory footprint and low prediction latency in multi-task models.These constraints motivate starting from a thin network and dynamically growing it during training.
- Motivation: Multi-task learning shares relevant information across prediction tasks, but unrestricted sharing can cause negative transfer between unrelated tasks.Negative transfer may worsen performance on both tasks when inadequate sharing occurs.
- Method: A greedy top-down criterion creates separate branches for unrelated tasks while penalizing model complexity.The method dynamically widens a thin network during training by creating new branches.
- Results: On CelebA, the method matches 90% facial-attribute accuracy with a model 90x more compact and 3x faster than VGG-16.The evaluation also reports preserved accuracy for simultaneous facial and clothing attribute prediction on DeepFashion.
2. Related Work
Prior multi-task vision methods typically use hand-designed sharing patterns, while related work on learned sharing and compression leaves a need for scalable, task-aware deep architectures.
- Multi-Task Learning: Earlier multi-task methods generally assume all tasks are related and suitable for joint training.Some methods address which tasks should share features, but are generally designed for shallow classification models.
- Deep Multi-Task Networks: Several deep multi-task methods use compact networks or cross-task connections, but rely on hand-designed shared layers and specialized branches.The cited approaches include HyperFace, UberNet, MultiNet, and Cross-ResNet.
- Learned Feature Sharing: Cross-stitching networks learn combinations of shared and task-specific representations, but their size grows linearly with the number of tasks.This growth creates a scalability issue as the task count increases.
- Model Compression and Acceleration: Deep convolutional models are computationally and memory intensive, motivating compression and acceleration methods such as distillation, factorization, pruning, and quantization.These methods target deployment settings with low memory resources or strict latency requirements.
- Person Attribute Classification: Person-attribute prediction methods commonly train separate classifiers, use fully shared networks, or use hand-designed branches that do not exploit differing attribute relatedness.The paper positions automated task-aware sharing against these approaches.
3. Methodology
The method constructs compact multi-task networks by initializing a thin model, then greedily widening and grouping task branches from the top down while penalizing complexity.
- Network representation: The network functional form chains layer computations, with each layer applying a parameterized linear operation followed by a non-linearity.For convolutional layers, P maps filter parameters into a matrix representing convolution as matrix multiplication.
- Training procedure: The algorithm begins with a thin-ω model, optionally initialized from a pretrained network, and iteratively trains, estimates affinity, selects branches, and widens the model.The procedure ends when no further branches are created, followed by convergence training of the fixed final network.
- Thin initialization: Thin models retain VGG-16 structure while limiting convolutional widths to min(ω, corresponding VGG-16 width) and fully connected widths to 2ω.For ω = 32, the illustrated thin network uses width 32 in convolutional layers and width 64 in fully connected layers.
- Thin initialization: SOMP selects a subset of pretrained filters layer by layer, then column-truncates subsequent weights to maintain consistent input dimensions.This provides a data-free initialization for the thin network despite its dimensional mismatch with the wider source model.
- Top-down widening: Widening creates junctions and branches associated with task subsets, proceeding recursively from the output layer toward lower layers as task groups become coarser.At the output layer branches correspond one-to-one with tasks; lower-layer branches may represent multiple tasks.
- Task grouping and width selection: At each layer, spectral clustering groups branches using task affinity, while a complexity-aware loss trades separation against branch-creation cost.The branch count is chosen by minimizing a loss whose creation penalty increases with pooling depth; larger α encourages more branches.
4. Experiments
Experiments show that adaptive branching produces compact, fast models with accuracy comparable to or better than strong baselines, while automatically grouping related tasks. Ablations indicate that similar-task grouping improves most task accuracies and SOMP initialization accelerates training, although fixed branch widths can limit difficult tasks.
- Datasets and metrics: The evaluation covers facial attributes on CelebA and clothing categories on DeepFashion, using classification accuracy and top-10 recall as metrics.CelebA contains 40 attribute classes; the models are also evaluated for clothing category recognition on DeepFashion.
- Comparison with baselines: Adaptive branching models achieve similar or better accuracy than state-of-the-art baselines while being more compact and faster.The comparison is reported in Tables 1 and 2 for the CelebA and DeepFashion test sets.
- Task grouping: Automatically learned groupings are often intuitive, clustering facial-hair, makeup, and age-related attributes without human intervention.Examples include “5-o-clock Shadow,” “Bushy Eyebrows,” and “No Beard,” plus age-related cues joined at fc6.
- Task grouping: Grouping similar tasks improves accuracy for most tasks relative to grouping dissimilar tasks.Manual and random shuffling were tested, with only manual-shuffling results reported because they were similar.
- Limitations and ablations: The fixed-width branch design may underserve intrinsically difficult tasks that require wider branches.The authors identify changing branch width as an interesting future direction after observing mixed effects within a large branch.
- Limitations and ablations: SOMP initialization makes the thin model converge faster and better under identical training conditions.The comparison uses the Baseline-thin-32 model on CelebA with and without SOMP initialization.
5. Conclusion
The paper proposes a compact multi-task architecture-learning method and reports compelling results on person attribute classification, with future extensions planned for incremental learning and domain adaptation.
- 5. Conclusion: The method learns compact multi-task deep neural network structures rather than relying on manually designed architectures.It starts with a thin network and expands it during training through multi-round branching that determines feature sharing while penalizing model complexity.
- 5. Conclusion: The approach achieved compelling results on person attribute classification.
- 5. Conclusion: Future work will adapt the approach to incremental learning and domain adaptation.