PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Relational Context for Multi-Task Dense Prediction

David Bruggemann, Menelaos Kanakis, Anton Obukhov, Stamatios Georgoulis, Luc Van Gool

arXiv:2104.13874v2cs.CV

TL;DR

Multi-task dense prediction needs ways to refine task predictions using cross-task context that reflects task relations. ATRC uses attention and neural architecture search to select context types for source-target task pairs; it outperforms comparable multi-modal distillation modules on NYUD-v2 and PASCAL-Context.

  • Problem

    Existing multi-modal distillation gates rely on source-task feature values and restrict each target pixel to its source counterpart, limiting task-interaction modeling and message-passing reach.

    Source: §1 · p. 1

  • Method

    ATRC adapts attention-based relational distillation and neural architecture search to select a context type for each source-target task pair.

    Source: §1 · p. 2

  • Results

    ATRC outperforms comparable multi-modal distillation modules on the NYUD-v2 and PASCAL-Context benchmarks.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The framework is presented as a drop-in module for refining supervised multi-task dense-prediction models.

    Source: §1 · p. 2

    On PASCAL-Context, multi-task approaches are less effective than single-task baselines; the paper attributes this possibly to the larger, more diverse task dictionary.

    Source: §4.3 · p. 7

Abstract

from arXiv · show

The timeline of computer vision research is marked with advances in learning and utilizing efficient contextual representations. Most of them, however, are targeted at improving model performance on a single downstream task. We consider a multi-task environment for dense prediction tasks, represented by a common backbone and independent task-specific heads. Our goal is to find the most efficient way to refine each task prediction by capturing cross-task contexts dependent on tasks' relations. We explore various attention-based contexts, such as global and local, in the multi-task setting and analyze their behavior when applied to refine each task independently. Empirical findings confirm that different source-target task pairs benefit from different context types. To automate the selection process, we propose an Adaptive Task-Relational Context (ATRC) module, which samples the pool of all available contexts for each task pair using neural architecture search and outputs the optimal configuration for deployment. Our method achieves state-of-the-art performance on two important multi-task benchmarks, namely NYUD-v2 and PASCAL-Context. The proposed ATRC has a low computational toll and can be used as a drop-in refinement module for any supervised multi-task architecture.

1. Introduction

The introduction motivates cross-task context for dense prediction and identifies limitations in existing multi-modal distillation. ATRC uses attention and automated context selection to adapt refinement to each source-target task pair.

1. Introduction

  • Multi-task learning shares representations, while visual tasks are often highly correlated and prior dense-prediction work uses soft-gated cross-task aggregation.The introduction situates its motivation within attention-based context methods in computer vision.Source: §1 · p. 1
  • Existing multi-modal distillation gates use source-task feature values and restrict each target pixel to information from its source counterpart.The introduction frames these as limitations in capturing task interactions and in the spatial reach of message passing.Source: §1 · p. 1
  • ATRC conditions cross-task gates on source-target pixel interdependence and enlarges the distillation receptive field to enable global message passing.The paper calls each pixel’s distillation receptive field its distillation context.Source: §1 · p. 2
  • ATRC adapts among global, local, T-label, S-label, and none context types for each source-target task pair.The selection is automated for the task connections in a given architecture.Source: §1 · p. 2
  • The module adapts scaled-dot product attention and neural architecture search, and is presented as a drop-in refinement module for supervised dense prediction tasks.The example architecture uses a shared backbone, task-specific heads, and ATRC modules to refine task predictions.Source: §1 · p. 2

2. Related Work

Related work spans parameter-sharing strategies, cross-task feature transfer, attention-based relational contexts, and automated architecture search. ATRC is positioned as a single-stage feature-refinement approach that selects context types across task pairs.

2. Related Work

  • Hard parameter sharing typically branches from shared early representations into task-specific heads, while soft parameter sharing transfers knowledge between separate networks through feature fusion.Feature fusion across network depth can be limited by computational cost; the paper describes its own fusion as operating at a single stage on high-level representations.Source: §2 · p. 2
  • PAD-Net and MTI-Net use source-feature-based, per-pixel gates, whereas ATRC models pairwise task similarities and permits global cross-task message passing.PAP and PSD instead refine task-specific maps using global and local self-attention, propagating affinity patterns across tasks.Source: §2 · p. 2
  • Prior attention methods define fixed relational contexts for single-task prediction, while ATRC selects among context types for each source-target task pair.The related work describes contexts ranging from global and local pixels to grouped-region prototypes.Source: §2 · p. 2
  • Figure 2 depicts ATRC as a multi-task refinement module with context-pooling blocks for task pairs and a search-stage mixture of context representations.After search convergence, a single context type is selected for each block.Source: Figure 2 · p. 3
  • Differentiable NAS optimizes operation-mixing weights over convex combinations, and ATRC uses this approach with a custom search space for context selection.The paper contrasts this with reinforcement-learning and evolutionary search approaches described as compute-intensive.Source: §2 · p. 3

3. Adaptive Task-Relational Context

ATRC refines multi-task predictions by pooling relational context for each source–target task pair, then searches for a context type suited to each pair. Its context options span global, local, and label-based attention, with a none operation available to block information flow.

3.1. Architecture Design

  • Each source–target task pair has a Context Pooling block; outputs for each target are fused with its original features to produce refined predictions.The example architecture uses a shared backbone and task-specific heads.Source: §3.1 · p. 3

3.2. Context Pooling Block

  • Context Pooling uses target features as queries and source features as keys and values, producing attention-weighted source features to augment the target.Attention weights represent pairwise feature similarity and can be interpreted as cross-task feature co-occurrence.Source: §3.2 · pp. 3–4

3.2.1 Global Context

  • Naive global attention lets each target pixel use every source pixel, but its attention-weight computation scales as O(L^2); a linearization scheme reduces complexity to O(L).The method uses a linear kernel and softplus in the query and key transformations.Source: §3.2.1 · p. 4

3.2.2 Local Context

  • Local context restricts attention to source pixels near each target pixel, using a 9 × 9 spatial neighborhood.This resembles convolution with a spatially adaptive filter.Source: §3.2.2 · p. 4

3.2.3 Label Context

  • T-label aggregates source features by target label regions, whereas S-label aggregates them by source label regions; regression regions use log-depth bins or normal-vector clustering.Auxiliary label-region maps are trained with ground-truth supervision and used to form prototypes; example attention maps focus on distinct label regions.Source: §3.2.3 · pp. 4–5

3.3. Automated Context Type Selection

  • Differentiable NAS selects among global, local, T-label, S-label, and none for each task-pair block, using Gumbel-Softmax to relax the sampled choices.To address high variance in converged architecture samples, the method regularizes entropy and fixes selections after they become low-entropy; discovered architectures are retrained from scratch for comparison.Source: §3.3 · p. 5

4. Experiments

Across NYUD-v2 and PASCAL-Context, ATRC improves on multi-task baselines and, with FPM, consistently outperforms MTI-Net. Experiments also find that useful context choices vary by task pair and adapt when ATRC is combined with other methods.

4.1. Setup

  • Experiments evaluate dense prediction on NYUD-v2 and PASCAL-Context with HRNet18, HRNet48, and ResNet-50 backbones, repeating runs five times.The datasets cover semantic segmentation, depth, surface normals, boundaries, and, on PASCAL-Context, human parts and saliency; metrics are task-specific.Source: §4.1 · pp. 5–6

4.2. Distillation Module Benchmarking

  • ATRC yields the best multi-task performance among the tested distillation modules; no single fixed relational context type dominates across tasks.Relational-context modules generally outperform the multi-task baseline, and all except local relational context beat the single-task baseline while using a lower computational footprint.Source: §4.2 · p. 6, Table 1 · p. 6

4.3. Comparison with State-of-the-Art

  • With FPM, ATRC consistently outperforms MTI-Net across the investigated comparisons, while ATRC improves on the multi-task baseline in all cases.The comparisons cover NYUD-v2 with HRNet18 and HRNet48 and PASCAL-Context with HRNet18; with FPM, ATRC distills at one scale while MTI-Net distills at four.Source: Table 2 · p. 7, Table 3 · p. 7, Table 4 · p. 7

4.4. Source Task Importance

  • Permutation importance on held-out data found self-attention most important for three tasks, semantic features more important for depth, and weak correlation between importance and search reliability (Pearson r=0.43).Boundary detection profited little from distillation; the three most important source-target connections had 100% search reliability.Source: §4.4 · pp. 7–8

4.5. Complementary Methods

  • On PASCAL-Context with a ResNet-50 architecture, ATRC improves performance of the ASPP-augmented network; across five search runs, selected local context rises from 0% without ASPP to 41.6% with ASPP.Source: §4.5 · p. 8, Figure 7 · p. 8
  • ATRC also improves performance when combined with BMTAS-branched multi-task networks.The experiments use a NAS-selected branched backbone rather than a fully shared one.Source: §4.5 · p. 8, Table 5 · p. 8

5. Conclusion

ATRC refines pixel-wise predictions by using inter- and intra-task relationships in contextual source features. It selects among four relational context types for each source-target pair and outperforms comparable multi-modal distillation modules on NYUD-v2 and PASCAL-Context.

5. Conclusion

  • ATRC uses scaled dot-product attention to enrich target-task features with contextual source-task features while explicitly accounting for task relationships.The module exploits both inter- and intra-task relationships to refine pixel-wise predictions.Source: §5 · p. 8
  • The framework defines four context types—global, local, T-label, and S-label—and customizes the type for each source-target task pair.The paper presents this as a framework for customized task-relational context descriptions.Source: §5 · p. 8
  • Experiments on NYUD-v2 and PASCAL-Context show ATRC outperforming comparable multi-modal distillation modules established in the literature.The conclusion describes the framework as promising for multi-task dense prediction.Source: §5 · p. 8

A. Training Details

Training used repeated runs, data augmentation, weighted task losses, and standard backbone optimization. Context-type search used entropy-regularized architecture parameters, followed by a vote-based final configuration.

A. Training Details

  • All experiments were repeated five times using the described training pipeline.The final configuration was retrained five times after the context-type search runs.Source: A. Training Details · pp. 10–11
  • Training augmentation included random scaling, cropping, horizontal flips, and color jitter, with depth and surface-normal labels corrected for the transformations.NYUD-v2 images were cropped to 425 × 560; PASCAL-Context images were padded to 512×512.Source: A. Training Details · p. 10
  • The multi-task network uses a weighted sum of task losses, with loss types and weights set separately for the prediction tasks.Weights were determined by logarithmic grid search using single-task networks; auxiliary predictions use cross-entropy and do not backpropagate into the main network.Source: A. Training Details · pp. 10–11
  • Backbones were ImageNet-pretrained and optimized with SGD using momentum 0.9 and weight decay 0.0005; training ran for 40,000 iterations with minibatches of 8.The learning rate followed a poly schedule, with an option for heads to use a tenfold higher initial rate than the backbone.Source: A. Training Details · p. 11
  • Context-type search updates architecture parameters with Adam and entropy regularization, then selects each block’s final type by majority vote across five search runs.The search uses single-level optimization, linearly anneals Gumbel-Softmax temperature from 1.0 to 0.05, and terminates a block’s search early when its top two architecture weights differ by more than 0.3.Source: A. Training Details · p. 11

B. Implementation Verification

The implementation checks compare reproduced single-task baselines with published figures and report an unsuccessful PSD reimplementation. The schematics describe learned transforms and different similarity functions across relational context types.

B. Implementation Verification

C. Relational Context Schematics

  • The context schematics show a learned 1×1 Conv-BN-ReLU transform, with different similarity functions for global context and the other contexts.Attention is shown for one target pixel, with normalization over all pixels of the attention map.Source: C. Relational Context Schematics · p. 11

D. Label Context: Regression Tasks

Regression label contexts are formed by task-specific discretization of continuous targets. Depth uses logarithmic bins and surface normals use a codebook-based triangular representation; classification is slightly worse than regression but satisfactory for both.

D. Label Context: Regression Tasks

  • Discretization of regression labels must be tailored to each task to obtain satisfactory performance.Continuous ground-truth values can be reformulated as classification labels by binning.Source: D. Label Context: Regression Tasks · p. 11
  • Depth values are divided into 40 logarithmic bins, and evaluation combines bin mean depths in log space using predicted scores as weights.The logarithmic bins account for naturally larger estimation error at larger depth values.Source: D. Label Context: Regression Tasks · pp. 11–12
  • Surface normals use 40 k-means codewords forming a Delaunay triangulation on the unit sphere, with three codeword weights reconstructing each normal.Evaluation selects the triangle with the greatest total probability, then uses its three codeword probabilities to reconstruct the normal.Source: D. Label Context: Regression Tasks · p. 12
  • Classification-based depth and surface-normal models perform slightly worse than regression models but are reported as satisfactory for both tasks.The comparison is shown in Fig. D-1, with a qualitative comparison in Fig. D-2.Source: D. Label Context: Regression Tasks · p. 12

E. Label Context: Performance Upper Bound

Ground-truth label regions provide an upper-bound test of label-context distillation, with results showing that region grouping can substantially improve multi-modal distillation.

E. Label Context: Performance Upper Bound

  • Regression and classification perform similarly for single-task depth and surface-normal estimation, supporting classification for forming label regions.Source: Figure D-1
  • Ground-truth label-region maps provide an upper bound for T-label and S-label context distillation.The experiment uses ground-truth regions instead of predicting the spatial maps from the input image.Source: E. Label Context: Performance Upper… · p. 12, Table E-1 · p. 12
  • Performance increases greatly for both T-label and S-label context, except for T-label context on boundary detection.Source: E. Label Context: Performance Upper… · p. 12, Table E-1 · p. 12
  • Classification achieves satisfactory qualitative results for depth and surface-normal estimation.Source: Figure D-2 · p. 13

F. Context Type Search Reliability

Repeated architecture searches yield measurable agreement in selected context types, while the authors caution that reliability is not required for effective search.

F. Context Type Search Reliability

  • Across five NYUD-v2 searches with an HRNet18 backbone, context selections show 71.2% overall agreement and Light’s kappa of 0.48.Light’s kappa averages Cohen’s kappa across run pairs and corrects for chance agreement.Source: F. Context Type Search Reliability · p. 13
  • Search reliability is not strictly necessary: when no context type dominates in performance, a valid search may be unreliable.A tie-breaking auxiliary objective, such as a resource loss, could help promote convergence in such cases.Source: F. Context Type Search Reliability · p. 13

G. How Important is Self-Attention in ATRC?

Self-attention is important for most investigated tasks, but cross-task distillation still provides a substantial performance boost without it.

G. How Important is Self-Attention in ATRC?

Related papers

Loading 2104.13874v2…