PaperlayerView exact revision on arXiv

Source-linked AI summary

MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning

Simon Vandenhende, Stamatios Georgoulis, Luc Van Gool

arXiv:2001.06902v5cs.CV

TL;DR

The paper addresses the gap that task affinities can vary with scale, while existing multi-task refinement methods typically use a fixed scale. MTI-Net models interactions at every scale, propagates distilled information across scales, and aggregates refined task features; experiments report better performance than single-task models alongside reduced memory and computation.

  • Problem

    Task interactions are often modeled at a fixed scale, despite evidence that high affinity between tasks at one scale is not guaranteed at other scales.

  • Method

    MTI-Net uses multi-scale multi-modal distillation, feature propagation from lower to higher scales, and feature aggregation to refine per-task predictions.

  • Results

    MTI-Net consistently outperforms its single-task counterparts by medium to large margins while reducing memory footprint and the number of calculations.

  • Takeaways & Limitations

    Modeling task interactions at multiple scales enables tasks to maximally benefit from each other within a multi-task network.

Abstract

from arXiv · show

In this paper, we argue about the importance of considering task interactions at multiple scales when distilling task information in a multi-task learning setup. In contrast to common belief, we show that tasks with high affinity at a certain scale are not guaranteed to retain this behaviour at other scales, and vice versa. We propose a novel architecture, namely MTI-Net, that builds upon this finding in three ways. First, it explicitly models task interactions at every scale via a multi-scale multi-modal distillation unit. Second, it propagates distilled task information from lower to higher scales via a feature propagation module. Third, it aggregates the refined task features from all scales via a feature aggregation unit to produce the final per-task predictions. Extensive experiments on two multi-task dense labeling datasets show that, unlike prior work, our multi-task model delivers on the full potential of multi-task learning, that is, smaller memory footprint, reduced number of calculations, and better performance w.r.t. single-task learning. The code is made publicly available: https://github.com/SimonVandenhende/Multi-Task-Learning-PyTorch.

1 Introduction and prior work

Multi-task learning can reduce computation and memory, but sharing information between unrelated tasks may cause negative transfer. MTI-Net addresses this by modeling task interactions separately across scales, propagating information between scales, and aggregating refined task features.

  • Motivation and prior work: Multi-task networks share layers, reducing memory footprint and avoiding repeated feature calculations while potentially improving performance through complementary task information.These benefits depend on whether associated tasks share complementary information or regularize one another.
  • Motivation and prior work: Joint learning can degrade single-task performance when information is shared between unrelated tasks, a problem known as negative transfer.Prior examples include improved surface-normal estimation coinciding with worse object detection and multi-task models underperforming single-task models.
  • Motivation and prior work: Existing approaches commonly predict all task outputs from one input-processing cycle without subsequently refining them using relationships among task predictions.The paper links this limitation to missed commonalities such as aligned depth discontinuities and semantic edges.
  • Motivation and prior work: Prior refinement methods model task interactions at only a fixed local or global scale, although affinity at one scale need not persist at another.Semantic segmentation and depth estimation can exhibit different relationships when viewed locally versus globally.
  • MTI-Net: MTI-Net explicitly models task interactions at separate scales, distills information at each scale, propagates it from lower to higher scales, and aggregates the resulting task features.The architecture starts from multi-scale backbone features and uses four considered scales in the described setup.
  • MTI-Net: The paper identifies multi-scale interactions as the basis for improvements over single-task counterparts while retaining multi-task learning’s computational advantages.Its stated contributions include dedicated distillation, feature-propagation, and feature-aggregation modules, alongside improved performance relative to single-task models.

2 Method

MTI-Net models task interactions separately across multiple receptive-field scales rather than assuming a single fixed-scale interaction pattern. It distills task information at each scale, propagates information from lower to higher scales, and aggregates refined task features for final predictions.

  • Feature aggregation: After scale-wise distillation and propagation, distilled task features from all scales are aggregated to produce the final task predictions.This aggregation is the final stage of the proposed multi-scale architecture.
  • Task interactions at different scales: Prior multi-modal distillation methods combine task-specific features at a fixed backbone scale, assuming one receptive field captures all relevant task interactions.MTI-Net challenges this assumption because task influence can vary with receptive-field size.
  • Task interactions at different scales: Affinity patterns can reach 65% pair correspondence, but their cross-task agreement depends on receptive field and therefore varies with scale.The analysis considers semantic segmentation, depth estimation, and edge detection on NYUD-v2.
  • Multi-scale multi-modal distillation: MTI-Net applies task-specific heads to multi-scale backbone features, producing task representations that can be distilled separately at each scale.This also provides deep supervision and models interactions for multiple receptive-field sizes.
  • Multi-scale multi-modal distillation: At each scale, spatial attention distills information from other tasks to refine the task features, with most filtering performed on low-resolution maps to limit overhead.The framework permits other distillation mechanisms, such as squeeze-and-excitation, in place of spatial attention.
  • Feature propagation across scales: The Feature Propagation Module harmonizes lower-scale task features, uses task-specific squeeze-and-excitation gating for refinement, adds residual updates, and passes them to the preceding higher scale.The shared representation lets each task select relevant features before refined features are combined with higher-scale backbone features.

3 Experiments

Experiments on NYUD-v2 and PASCAL evaluate MTI-Net’s multi-scale design through controlled ablations, comparisons, and computational analysis. Across these evaluations, the model improves multi-task performance over single-task and competing multi-task baselines while reducing computational demands in reported comparisons.

  • Experimental setup: Experiments use PASCAL and NYUD-v2 with FPN and HRNet backbones, four output scales, and pre-trained ImageNet weights.The scales are 1/4, 1/8, 1/16, and 1/32; task-specific heads use two residual blocks.
  • Ablation studies: NYUD-v2 ablations show performance rising from −1.71% for the MTL baseline to +6.40% with feature propagation and +10.91% with auxiliary tasks.The comparisons are against the set of single-task models using an HRNet-18 backbone.
  • Ablation studies: PASCAL ablations show the MTL baseline decreases performance by −4.26% and −3.70%, whereas MTI-Net reaches +3.35% on the small task set and +2.74% across all five tasks.Adding edges reaches +3.98%, adding normals yields +2.69%, and adding both remains stable at +3.36%; normals show a small decrease when all tasks are learned together.
  • Ablation studies: Increasing the number of backbone scales gradually improves NYUD-v2 performance, supporting the proposed multi-scale interaction design.Using a single scale reduces MTI-Net to a PAD-Net-like architecture.
  • Comparison with the state-of-the-art: On PASCAL, MTI-Net achieves +1.36% versus ASTMT’s −0.87% with the same R50-FPN backbone and +2.74% versus PAD-Net’s −3.08%.With R18-FPN, MTI-Net also reaches +0.29% versus ASTMT’s −0.08% despite ASTMT using a deeper DLv3+ backbone.
  • Computational analysis: On PASCAL, MTI-Net improves all three reported computational and performance metrics over single-task models and outperforms ASTMT in FLOPS and multi-task performance.The model performs most computations at smaller scales while using only C channels at the higher 1/4 scale; ASTMT requires a separate forward pass per task.

4 Conclusion

MTI-Net models task interactions at multiple scales through dedicated modules, enabling tasks to benefit from one another. The resulting multi-task models use fewer resources and consistently outperform single-task counterparts.

  • MTI-Net models task interactions at multiple scales to enable tasks to maximally benefit each other.
  • The architecture adds multi-scale multi-modal distillation, cross-scale feature propagation, and feature aggregation modules.
  • The model delivers a smaller memory footprint, fewer calculations, and better performance than single-task learning.
  • MTI-Net models consistently outperform their single-task counterparts by medium to large margins.

A.1 Difference with Cross-Stitch Networks

MTI-Net differs from cross-stitch networks in where and how it models task interactions, and in how efficiently it scales with the number of tasks.

  • MTI-Net fuses task features near the output, where they contain more disentangled task information, rather than interacting during encoding.
  • Cross-stitch networks distill task information sequentially at a local scale, whereas MTI-Net globally fuses information across all scales in parallel.
  • Global multi-scale fusion enables long-term relationships and sufficient context for dense prediction tasks.
  • Cross-stitch model size grows linearly with the number of tasks, while MTI-Net is closer to single-task model size.

A.2 Training setup

The experiments use HRNet and FPN as multi-scale backbones, with backbone-specific decoding choices in the feature aggregation unit.

  • The training setup considers two multi-scale backbone networks: HRNet and FPN.
  • For HRNet, feature aggregation uses bilinear upsampling, concatenation, and two convolutional layers to decode multi-scale features.
  • For FPN, the feature aggregation module decodes multi-scale features as in panoptic feature pyramid networks.

NYUD-v2

For NYUD-v2, the stated augmentation and optimization setup uses random scaling, horizontal flips, Adam, and polynomial learning-rate decay.

  • RGB and depth images are randomly scaled by ratios in {1, 1.2, 1.5} and randomly horizontally flipped.
  • The model is trained for 80 epochs with Adam, an initial learning rate of 1e-4, and batches of size 6.
  • A poly learning-rate decay scheme is used during training.

PASCAL

The PASCAL experiments use established training settings for fair comparison, while MTI-Net uses Adam and reuses the previously selected loss weights.

  • PASCAL: Single-task and multi-task baselines use shared optimization settings, including momentum 0.9, batch size 8, poly learning-rate decay, learning rate 0.01, and weight decay 1e-4.These hyperparameters match those used in the comparison code base.
  • PASCAL: MTI-Net uses Adam with an initial learning rate of 1e-4 and reuses earlier loss weights for its initial task predictions.

A.3 Extra experiments on PASCAL

The ResNet-18 FPN ablation supports MTI-Net across backbone settings, while auxiliary-task effects depend on the auxiliary task and label quality.

  • A.3 Extra experiments on PASCAL: MTI-Net improves over single-task models with a ResNet-18 FPN backbone for both the small and complete task sets.The reported improvements are +2.77% for the small set and +3.84% for the complete set.
  • A.3 Extra experiments on PASCAL: Adding edge detection as an auxiliary task raises the small-task improvement from +2.77% to +3.61%.
  • A.3 Extra experiments on PASCAL: Adding surface normals reduces the small-task improvement from +2.77% to 2.52%.Including both edge detection and surface normals changes the improvement from 3.61% to 3.06%.
  • A.3 Extra experiments on PASCAL: The authors attribute the weaker surface-normal result to approximate, noisy labels obtained through distillation.
  • A.3 Extra experiments on PASCAL: The ablation concludes that MTI-Net can be combined with various backbone architectures.

A.4 Extra experiments on NYUD-v2

The NYUD-v2 supplementary experiments extend the main evaluation with HRNet18 metrics, ResNet-18 FPN ablations, qualitative predictions, and task-performance views.

  • A.4 Extra experiments on NYUD-v2: The supplementary NYUD-v2 section provides additional HRNet-18 ablation results and an experiment using a ResNet-18 FPN backbone.
  • A.4 Extra experiments on NYUD-v2: The supplementary tables identify auxiliary tasks in brackets and include both HRNet18-V2 and ResNet-18 FPN configurations.
  • A.4 Extra experiments on NYUD-v2: Table S2 adds depth-estimation and semantic-segmentation metrics for an HRNet18 backbone beyond those shown in the main paper.
  • A.4 Extra experiments on NYUD-v2: The ResNet-18 FPN ablation reports similar findings to HRNet-18, including significant improvement over single-task models and benefits from auxiliary tasks.
  • A.4 Extra experiments on NYUD-v2: Figure S1 presents qualitative semantic and depth predictions from the HRNet-48 model on NYUD-v2 test images.Its quantitative results were reported separately in Table 6.
Loading 2001.06902v5…