PaperlayerView exact revision on arXiv

Source-linked AI summary

Attentive Single-Tasking of Multiple Tasks

Kevis-Kokitsi Maninis, Ilija Radosavovic, Iasonas Kokkinos

arXiv:1904.08918v1cs.CV

TL;DR

Multi-task networks can suffer task interference when information useful for one task conflicts with another, lowering per-task performance. This paper executes one task at a time, adapts shared features with task attention and residual adapters, and regularizes task gradients adversarially. Across dense labelling problems, it attains or surpasses single-task performance while reducing parameters and trading computation against multi-task accuracy.

  • Problem

    Multi-task networks can lose per-task performance because conflicting task objectives make shared representations interfere as more tasks are added.

  • Method

    The method executes one task at a time and uses task-specific feature modulation, residual adaptation, and adversarial training to separate task-specific from shared processing.

  • Results

    The method attains and can surpass single-task performance while reducing parameters, with modulation and adversarial training improving performance compared with standard multi-tasking.

  • Takeaways & Limitations

    Single-tasking multiple tasks provides a shared network that can focus on the task at hand while preserving or improving multi-task accuracy.

  • Takeaways & Limitations

    Task-specific modulation prevents sharing computation in subsequent layers across all tasks, increasing the number of multiply-adds.

Abstract

from arXiv · show

In this work we address task interference in universal networks by considering that a network is trained on multiple tasks, but performs one task at a time, an approach we refer to as "single-tasking multiple tasks". The network thus modifies its behaviour through task-dependent feature adaptation, or task attention. This gives the network the ability to accentuate the features that are adapted to a task, while shunning irrelevant ones. We further reduce task interference by forcing the task gradients to be statistically indistinguishable through adversarial training, ensuring that the common backbone architecture serving all tasks is not dominated by any of the task-specific gradients. Results in three multi-task dense labelling problems consistently show: (i) a large reduction in the number of parameters while preserving, or even improving performance and (ii) a smooth trade-off between computation and multi-task accuracy. We provide our system's code and pre-trained models at http://vision.ee.ethz.ch/~kmaninis/astmt/.

1. Introduction

Universal vision systems must support many visual tasks, but sharing one network can reduce per-task performance as tasks accumulate. This motivates multi-task architectures that address both breadth and task-specific accuracy.

  • 1. Introduction: Real-world applications such as autonomous driving and human-computer interaction require multiple visual tasks.Examples include pedestrian detection, velocity estimation, traffic-sign reading, identity recognition, pose, and face and hand tracking.
  • 1. Introduction: Multi-task networks handle increasing numbers of tasks, but single-task networks remain common when per-task performance is critical.The supplied passage frames this as a practical response to observed multi-task performance limits.
  • 1. Introduction: Adding more tasks to one network can cause performance drops, while stronger backbones improve multi-task performance without eliminating the per-task gap.The passage attributes the drop to limited network capacity and contrasts multi-task performance with single-task performance using the same backbone.

Ariel AI, UCL

Task interference arises because information useful for one task can be nuisance information for another. The paper addresses this by executing one task at a time and adapting a shared network’s features and gradients to the selected task.

  • Ariel AI, UCL: Later network layers become more task-adapted and disentangled, whereas early-layer features remain more similar across tasks.The representation visualization uses a common PCA basis across tasks and layers.
  • Ariel AI, UCL: Task interference reflects conflicting requirements for invariance and sensitivity across tasks.For pose estimation and object detection, detailed pose can be useful to one task but nuisance information to the other.
  • Ariel AI, UCL: Single-tasking executes one task at a time while retaining the same shared backbone across tasks.The network changes its behavior according to the executed task and uses task-appropriate features.
  • Ariel AI, UCL: Task attention adapts network behavior through task-specific feature modulation and residual adaptation.The two mechanisms respectively modulate activity and append residual adapter blocks that refine shared representations.
  • Ariel AI, UCL: Adversarial training makes task gradients statistically indistinguishable to prevent any task from overwhelming shared-representation training.The task-adversarial loss is minimized with double backpropagation and promotes compartmentalization between shared and task-specific blocks.

2. Related Work

Related work shows that joint learning can provide complementary information but can also harm tasks with unrelated or conflicting objectives. Existing responses include loss balancing, gradient manipulation, task-specific representations, compartmentalization, and attention.

  • 2. Related Work: Jointly learning related task pairs can yield fruitful results through complementary information or regularization.Examples include detection with classification or segmentation, and monocular depth with segmentation.
  • 2. Related Work: Joint learning can harm tasks without a direct relationship, including combinations of low-, mid-, and high-level vision tasks.The cited examples report improvement in one task occurring alongside deterioration in another.
  • 2. Related Work: Prior methods address interference by calibrating task losses or homogenizing task gradients.GradNorm normalizes task gradients, while later work extends gradient homogenization through adversarial training.
  • 2. Related Work: Task-specific representations can prevent gradient spillover but may increase complexity linearly with the number of tasks.Related approaches also remove gradient components that would increase losses for previous tasks.
  • 2. Related Work: Soft, learnable task-specific feature masking offers compartmental operation without changing the network’s weight matrix.The approach shares similarities with attention mechanisms, including channel modulation by Squeeze-and-Excitation networks.
  • 2. Related Work: The single-task architecture uses Deeplab-v3+ with an SE-ResNet backbone and task-specific SE layers.SE modules appear in bottleneck blocks of both encoder and decoder.

3. Attentive Single-Tasking Mechanisms

The proposed mechanisms let tasks use a shared representation differently through soft channel modulation and task-specific residual adapters. These mechanisms aim to reduce interference while adding only limited task-specific resources.

  • 3. Attentive Single-Tasking Mechanisms: The method learns a shared representation while allowing each task to use it differently when constructing task-specific features.This targets efficiency while addressing conflicts between unrelated task objectives.
  • 3.1. Task-specific feature modulation: Hard channel masking can prevent task A from using task B’s features and eliminate task A’s gradient through those channels.The minimal example assumes different channel subsets are better suited to different tasks.
  • 3.1. Task-specific feature modulation: Squeeze-and-Excitation replaces hard masks with learned, image-adaptive channel gating whose parameters are task-dependent.Global average pooling and a sigmoid-producing fully connected layer yield differentiable modulation signals.
  • 3.2. Residual Adapters: Residual adapters append task-specific subnetworks that adapt and refine shared features through residual operations.They complement feature modulation by processing task-specific information alongside the generic backbone representation.
  • 3.2. Residual Adapters: Adapters typically use a small fraction of the shared network’s parameter and computation budget while improving accuracy substantially.The task-specific budget is used together with the shared backbone budget.
  • 3.2. Residual Adapters: Task-specific batch-normalization layers add few parameters without increasing computational cost.They are used with disentangled computation graphs employing feature modulation and/or residual adapters.

4. Adversarial Task Disentanglement

The method separates shared and task-specific processing by adapting features per task and adversarially enforcing task-indistinguishable gradients at the shared backbone. This preserves shared memory and computation while preventing any task from dominating shared parameters.

  • Task-specific adaptation: Task-specific modulation or residual adapters give each task separate modelling capacity alongside a shared representation.These mechanisms enhance or suppress neuronal activity or fuse task-specific residual processing with generic backbone features.
  • Adversarial training: The discriminator-driven gradient reversal is controlled by λ ∈[0, 1], which sets the amount of negative gradient returned to the network.The resulting shared-feature gradients have the same distribution irrespective of task, preventing domination associated with higher loss magnitude.
  • Gradient disentanglement: The method enforces statistically indistinguishable gradients from different tasks at shared parameters, separating shared and task-specific processing.The constraint is intended to keep the shared backbone serving all tasks equally well rather than being disrupted by larger gradients from one task.
  • Gradient disentanglement: Double back-propagation exposes the gradient sent from task t to shared layer l by unfolding its computation graph, enabling backpropagation through that gradient.The exposed gradient is denoted g_t^(l) in the method description.
  • Adversarial training: A discriminator is trained to classify the source task of gradients while sign negation makes the task classifier fail.The optimization is min_wD max_wN L(D(g_t(w_N), w_D), t), where t labels the source task.

5. Experimental Evaluation

Experiments evaluate the approach on dense prediction tasks across PASCAL, NYUD, and FSV, including ablations, backbone variations, resource analysis, and qualitative comparisons. Task modulation and adversarial training recover or improve multi-task performance while reducing resource demands relative to single-tasking and standard multi-tasking.

  • Datasets and setup: The evaluation covers dense prediction on PASCAL, NYUD, and FSV, using tasks including segmentation, depth, surface normals, saliency, and edge detection.PASCAL is the main benchmark; NYUD and FSV provide additional validation datasets.
  • Ablations: -6.6% average performance per task occurs for the R-26 SE multi-tasking baseline when all tasks share one network.The baseline connects task-specific 1 × 1 convolutional classifiers at the network end.
  • Ablations: -1.4% and -0.6% average drops are obtained with residual adapters and SE modulation, respectively, compared with -6.6% for the shared baseline.Separating task computation during learning immediately recovers performance; decoder-only SE reaches -1.4% with one shared encoder pass.
  • Ablations: -4.4% versus -6.6% average performance drop shows that adversarial training improves standard multi-tasking without extra inference parameters or computation.The discriminator classifies gradient sources and is used only during training.
  • Generalization: The method consistently improves standard multi-tasking across backbone depths and transfers to NYUD and FSV, where performance recovery is observed after adding modulation and adversarial training.Additional datasets show larger improvements than PASCAL, while SEA improvements are observed regardless of backbone capacity or depth.
  • Resource analysis: The most accurate variant outperforms single-tasking baselines using only a fraction of their parameters, while adversarial-training gains add only a small training-time discriminator overhead.Modulation improves performance with slight resource increases, becoming more costly when applied in the encoder and decoder.
  • Resource analysis: Task-specific adapters use a small fraction of the shared-network budget and substantially improve accuracy, but task-specific outputs prevent sharing computation in subsequent layers.The resulting increase is primarily in multiply-adds rather than the adapter blocks’ own memory and computational complexity.

6. Conclusions

The method attains or surpasses single-task performance by executing one task at a time and focusing the network through task-specific feature modulation and adaptation. The authors identify interleaved task execution as a direction for future research.

  • The method attains or surpasses single-task performance when multi-task networks execute one task at a time.
  • Task-specific feature modulation and adaptation let the network focus on the task at hand.
  • Future work will explore task attention determined by the current operation and interleaved low-level and high-level task execution.

Appendix

The appendix provides additional ablations, baselines, mobile-backbone results, and training and testing details. It also states that code will be published.

  • Appendix A reports ablated results using different backbone architectures on NYUD and FSV.
  • Appendix B compares UberNet-type networks with and without the proposed modulation and adversarial training.
  • Appendix C reports results using MobileNet as the backbone to highlight potential mobile-phone applications.
  • Appendix D gives technical details for training and testing, and the authors state that code will be published.

Appendix A: More results on NYUD and FSV

Additional NYUD and FSV experiments show that the method remains effective across backbone architectures. Modulation and adversarial training recover standard multi-tasking losses in UberNet-type settings.

  • Standard multi-task learning causes a significant performance drop in UberNet experiments, recovered with modulation and adversarial training.
  • The reimplemented UberNet uses a shared backbone and lightweight task-specific heads with skip connections and 1 × 1 convolutions.The authors contrast these heads with their Deeplab-v3+ ASPP decoder.

Appendix C: MobileNet-v2 backbone for multiple tasks

The MobileNet-v2 experiments adapt the framework for lightweight multi-task networks intended for mobile applications. SE-based modulation recovers performance lost by standard multi-task learning, while MobileNet modulation reaches R-50 results with fewer resources.

  • The framework targets mobile applications by shipping a single small parameter set for many tasks.
  • MobileNet with the proposed modulation reaches R-50 results for standard multi-tasking using fewer parameters and less computation.
  • SE modulation recovers the performance lost through standard multi-task learning on PASCAL.
  • MobileNetv2 is adapted with depth-wise convolutions and ReLU6 activations for multi-task learning.An SE-MobileNet variant is pre-trained on ImageNet and fine-tuned for multi-task learning.

Appendix D: Implementation Details

The implementation uses task-specific losses and weighting, standardized optimization settings, and task-aware gradient handling. It also records task-specific loss definitions and a stated limitation for albedo prediction.

  • Optimization: Optimization uses synchronized SGD with momentum 0.9, weight decay 1e-4, initial learning rate 0.005, and a polynomial learning-rate schedule.Models use batch size 8 and 512 × 512 inputs on a single GPU, with dataset-specific epoch counts.
  • Loss weighting: Grid-search selects loss weights because optimal task weights are highly imbalanced across tasks.The implementation reports that edge detection’s optimal loss weight is 50 times higher than semantic segmentation’s.
  • Optimization: Multi-task training divides the shared-layer learning rate by T because the shared layers receive T updates for each mini-batch.This adjustment is applied for all multi-task experiments.
  • Loss weighting: Losses combine task-specific terms wt · Lt with discriminator loss wd · Ld, averaged over the number of predicted samples.The implementation explicitly defines wt as the task-loss weight and wd as the discriminator-loss weight.
  • Task objectives: Task losses use task-specific objectives, including cross-entropy for segmentation, L1 losses for surface normals and albedo, and a combined L1-smoothness loss for monocular depth.Surface-normal training additionally uses unit-vector normalization, while depth experiments report better quantitative and qualitative results with the smoothness term.
  • Task objectives: Albedo prediction is limited because the architecture outputs at one-quarter the input resolution, preventing capture of tiny details.An architecture specifically suited to albedo is outside the scope of this work.
  • Adversarial training: The discriminator has two 1 × 1 convolutional layers and a ReLU, while normalized task-loss gradients improve training stability.Larger discriminator depth did not improve results, and the discriminator loss weight is wd = 0.1.
Loading 1904.08918v1…