PaperlayerView exact revision on arXiv

Source-linked AI summary

NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reduction

Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, Alan L. Yuille

arXiv:1801.08297v4cs.CVcs.LG

TL;DR

Existing multi-task CNNs often use heuristic feature-sharing structures, raising questions about how to share representations across tasks with differing feature needs. The paper proposes NDDR, which concatenates same-resolution task features and applies discriminative dimensionality reduction through standard CNN components. Experiments across task sets and architectures report promising performance and generalizability, while ablations indicate easy training and robustness to hyperparameters.

  • Problem

    MTL CNN feature sharing is commonly chosen heuristically, although improper sharing can degrade task performance and concatenation creates channel dimensions incompatible with later layers.

  • Method

    NDDR concatenates same-resolution task features and uses 1 × 1 convolution, batch normalization, and weight decay to learn task-specific discriminative embeddings within an end-to-end CNN.

  • Results

    Experiments on various CNN structures and task sets demonstrate promising performance and desirable generalizability; one comparison exceeds the sluice network by around 3.8% on surface-normal prediction and 1.1%-1.2% on both semantic-segmentation metrics.

  • Takeaways & Limitations

    The existing-component design supports extending NDDR across state-of-the-art CNN architectures in a plug-and-play manner, with ablations indicating easy training and hyperparameter robustness.

  • Takeaways & Limitations

    The paper hypothesizes that features from multiple task descriptors contain additional discriminative information, and notes that NDDR weight magnitudes may require a larger learning rate.

Abstract

from arXiv · show

In this paper, we propose a novel Convolutional Neural Network (CNN) structure for general-purpose multi-task learning (MTL), which enables automatic feature fusing at every layer from different tasks. This is in contrast with the most widely used MTL CNN structures which empirically or heuristically share features on some specific layers (e.g., share all the features except the last convolutional layer). The proposed layerwise feature fusing scheme is formulated by combining existing CNN components in a novel way, with clear mathematical interpretability as discriminative dimensionality reduction, which is referred to as Neural Discriminative Dimensionality Reduction (NDDR). Specifically, we first concatenate features with the same spatial resolution from different tasks according to their channel dimension. Then, we show that the discriminative dimensionality reduction can be fulfilled by 1x1 Convolution, Batch Normalization, and Weight Decay in one CNN. The use of existing CNN components ensures the end-to-end training and the extensibility of the proposed NDDR layer to various state-of-the-art CNN architectures in a "plug-and-play" manner. The detailed ablation analysis shows that the proposed NDDR layer is easy to train and also robust to different hyperparameters. Experiments on different task sets with various base network architectures demonstrate the promising performance and desirable generalizability of our proposed method. The code of our paper is available at https://github.com/ethanygao/NDDR-CNN.

1. Introduction

The paper addresses heuristic feature sharing in multi-task CNNs by proposing layerwise fusion through Neural Discriminative Dimensionality Reduction. NDDR concatenates task features, learns discriminative embeddings, and preserves compatibility with subsequent CNN layers.

  • Motivation: MTL CNNs commonly share all convolutional layers and split at fully connected layers, despite tasks potentially requiring different low- and mid-level features.Prior work also reports that improper sharing can degrade some or all task performances, while exhaustive structure search is infeasible.
  • Motivation: Task-specific features at matching spatial resolutions can be concatenated across channels, but this expands the channel count and violates subsequent layers’ fixed input requirements.The proposed approach treats hierarchical features from different task networks as potentially complementary representations.
  • NDDR: NDDR formulates discriminative dimensionality reduction using 1 × 1 convolution, batch normalization, and weight decay within an end-to-end CNN.The transformation projects concatenated features into task-specific embeddings while retaining discriminative information and satisfying downstream channel requirements.
  • NDDR: The proposed general-purpose MTL structure combines existing CNN components and is designed for extensibility across state-of-the-art architectures in a plug-and-play manner.The paper presents NDDR-CNN and its variant NDDR-CNN-Shortcut as layerwise feature-fusing structures.

2. Related Works

Related work spans multi-task feature sharing, discriminative dimensionality reduction, and 1 × 1 convolution. The paper positions NDDR as a novel combination of established CNN components for multi-task feature fusion.

  • Multi-task learning: Prior MTL methods explore feature sharing at fully connected layers, automatic layerwise splitting, tensor-decomposition initialization, loss weighting, and trainable feature-combination scalars.These approaches address how task representations or supervision should be shared and combined.
  • Discriminative dimensionality reduction: Discriminative dimensionality reduction seeks low-dimensional embeddings that retain discriminative information while reducing computational and storage costs.Examples include linear discriminant analysis and low-rank metric learning.
  • CNN components: Although 1 × 1 convolution is established in CNN architectures, NDDR combines it with batch normalization and weight decay to learn multi-task feature embeddings.The paper explicitly distinguishes this combination from claiming 1 × 1 convolution itself as novel.

3. Methodology

The method replaces heuristic split-style multi-task CNN design with layerwise feature fusing through NDDR layers, which learn task-specific discriminative embeddings from concatenated features.

  • NDDR-CNN: NDDR-CNN automatically learns layerwise feature-fusing structures instead of splitting task networks at a fixed convolutional layer.It connects multiple single-task networks through NDDR layers inserted at several network levels.
  • NDDR Layer: Task-specific features with matching spatial resolution are concatenated across channels, then reduced to the channel size required by subsequent CNN layers.The reduction learns a discriminative embedding for each task.
  • NDDR Layer: 1×1 convolution implements the discriminative transformation, while batch normalization and ℓ2 weight decay constrain inputs and transformation weights for stable learning.These operations let the NDDR layer train end-to-end using task-specific losses and weight-decay loss.
  • NDDR-CNN: NDDR-CNN connects original single-task networks at multiple levels, and its K-task construction concatenates features before 1×1 convolutional dimensionality reduction.The layer uses C filters over KC input channels when each task output has C channels.
  • Implementation: The design uses common CNN operations, enabling extension to VGG, ResNet, and DenseNet in a plug-and-play manner with minimal architectural modification.The baseline networks are connected by applying NDDR layers at the ends of stages or blocks.

4. Ablation Analysis

The ablation study evaluates NDDR-layer initialization, learning-rate scaling, and pretrained-weight choices on semantic segmentation and surface normal prediction. Diagonal initialization, enlarged NDDR learning rates, and target-task single-task weights are favored by the reported results.

  • Ablation setup: The ablation study examines NDDR initialization, learning-rate scaling, and pretrained models using semantic segmentation and surface normal prediction on NYU v2.The dataset uses 795 training images and 654 validation images, with 40 semantic classes.
  • Initializations for NDDR Layers: Diagonal initialization outperforms Xavier initialization, while varying (α, β) has little effect; subsequent experiments use (α, β) = (0.9, 0.1).The tested diagonal initializations range from (1, 0) to (0, 1).
  • Pretrained Models: The authors compare weights pretrained on a general task with weights finetuned on corresponding target single tasks.The comparison includes pretrained Deeplab-VGG-16 weights for semantic segmentation on Pascal VOC 2012 and target single-task weights.
  • Learning Rates for NDDR Layer: Larger NDDR-layer learning rates improve performance, leading the authors to use 100 times the base learning rate in later experiments.The scale is defined relative to the learning rate for the remaining network layers.
  • Pretrained Models: Finetuned weights from the corresponding target single tasks perform better than the other pretrained models in the reported comparison.The authors interpret adding NDDR layers as enlarging the capability of the converged original networks.

5. Experiments

Experiments evaluate NDDR-CNN across task sets and network architectures against single-task, heuristic multi-task, cross-stitch, and sluice baselines. NDDR-CNN consistently performs strongly, with improvements reported for scene-labeling, surface-normal, and age-estimation tasks.

  • Experimental setup: The comparisons include single-task, heuristic multi-task, cross-stitch, and sluice-network baselines.The heuristic multi-task baseline shares all convolutional layers and splits after the last convolutional layer.
  • Experimental setup: Experiments cover semantic segmentation and surface normal prediction on NYU v2, plus age and gender classification on IMDB-WIKI.The study uses diverse pixel-level and image-level task sets to assess general-purpose multi-task learning.
  • ResNet-101 results: NDDR-CNN consistently outperforms baseline and state-of-the-art results on ResNet-101 after adding five NDDR layers.The reported modification leaves the 101-layer backbone largely unchanged apart from the five NDDR layers.
  • VGG-16-Shortcut results: On VGG-16-Shortcut, NDDR-CNN exceeds the sluice network by 3.1% on “within 11.25◦” and by 1.0%-1.5% on the two segmentation metrics.All methods improve over vanilla VGG-16, with larger improvements for the proposed method than its counterpart.
  • Age and gender results: For age estimation, NDDR-CNN improves Mean AE by approximately 5.9% and Median AE by approximately 11.4%, while gender classification is comparable with cross-stitch.The paper attributes the smaller gender benefit to sufficient labeled samples for both genders in this two-class task.

6. Conclusions

The proposed NDDR-CNN structure fuses task-specific features layerwise through NDDR, combining existing CNN components for interpretable, extensible multi-task learning. Experiments and ablations report promising performance, generalizability, trainability, and hyperparameter robustness.

  • NDDR concatenates same-resolution task-specific features and learns a discriminative embedding for each task while matching subsequent layers’ channel requirements.
  • The NDDR layer combines existing CNN components, supports end-to-end training, extends to state-of-the-art architectures in a plug-and-play manner, and generalizes cross-stitch and sluice networks.
  • Detailed ablations show that NDDR is easy to train and robust to different hyperparameters, while experiments across CNN structures and task sets demonstrate promising performance and generalizability.

Appendix

The appendix reports additional experiments on AlexNet and a cross-stitch hyperparameter ablation on VGG-16, with corresponding result and ablation tables.

  • The appendix includes semantic segmentation and surface normal prediction experiments using an AlexNet backbone.
  • It also includes cross-stitch ablation analysis on VGG-16 to verify the hyperparameters used in the main text.
  • Table A1 presents results for semantic segmentation and surface normal prediction on AlexNet.
  • Table A2 presents cross-stitch ablation analysis on VGG-16, examining the selected fusion hyperparameters and fuse-layer learning rate.

A1. Semantic Segmentation and Surface Normal Prediction on AlexNet

On AlexNet, the study evaluates semantic segmentation and surface normal prediction using FCN32s and the cross-stitch paper’s hyperparameters. NDDR-CNN outperforms the cross-stitch and sluice networks in these experiments.

  • The experiments use AlexNet with FCN32s and the same hyperparameters as the cross-stitch network paper for semantic segmentation and surface normal prediction.
  • NDDR-CNN outperforms both the cross-stitch network and the sluice network on AlexNet.

A2. Ablation Analysis for the Cross-Stitch Network on VGG-16

The VGG-16 ablation evaluates whether the cross-stitch hyperparameters used for comparison are optimal on another backbone. The analysis supports their use for fair comparative evaluation.

  • The VGG-16 ablation tests cross-stitch hyperparameters on a network backbone different from AlexNet to assess fair comparisons with the state of the art.
Loading 1801.08297v4…