PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Cross Residual Learning for Multitask Visual Recognition

Brendan Jou, Shih-Fu Chang

arXiv:1604.01335v2cs.CVcs.AIcs.MM

TL;DR

Visual recognition networks lack a way to model relationships among closely coupled tasks. The paper introduces cross-residual connections for multitask learning; its network uses >40% less model memory while achieving competitive or better detection performance, including about 10.4% better performance than a standard multitask residual network.

  • Problem

    Residual networks do not model relationships among closely coupled recognition tasks, while multimedia systems often treat related concept-detection tasks independently.

    Source: §1 · p. 1

  • Method

    Cross-residual learning adds cross-task connections that mix information among related task representations in a multitask network.

    Source: §3 · p. 3

  • Results

    The network uses >40% less model memory while matching or improving on specialized single-task networks and outperforming standard multitask residual networks by about 10.4%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Cross-residual multitask networks can jointly detect noun, adjective, and adjective-noun visual concepts with competitive or better performance than specialized networks.

    Source: §1 · p. 2, §6 · p. 8

    The authors do not claim significantly higher accuracy than single-task networks; the evaluated multitask network used only two cross-residual layers, and adding layers or branching earlier could increase parameter cost.

    Source: §5.2.2 · p. 7

Abstract

from arXiv · show

Residual learning has recently surfaced as an effective means of constructing very deep neural networks for object recognition. However, current incarnations of residual networks do not allow for the modeling and integration of complex relations between closely coupled recognition tasks or across domains. Such problems are often encountered in multimedia applications involving large-scale content recognition. We propose a novel extension of residual learning for deep networks that enables intuitive learning across multiple related tasks using cross-connections called cross-residuals. These cross-residuals connections can be viewed as a form of in-network regularization and enables greater network generalization. We show how cross-residual learning (CRL) can be integrated in multitask networks to jointly train and detect visual concepts across several tasks. We present a single multitask cross-residual network with >40% less parameters that is able to achieve competitive, or even better, detection performance on a visual sentiment concept detection problem normally requiring multiple specialized single-task networks. The resulting multitask cross-residual network also achieves better detection performance by about 10.4% over a standard multitask residual network without cross-residuals with even a small amount of cross-task weighting.

1. INTRODUCTION

The introduction frames visual concept detection as a setting where related tasks are often handled independently despite shared semantic and visual structure. It proposes cross-residual learning and a multitask network to learn across those tasks.

1. INTRODUCTION

  • Object recognition, affect prediction, and adjective-noun pair detection are related but are often treated independently, despite adjective-noun pairs linking objects with affective qualities.Examples include “happy girl,” “misty woods,” and “good food.”Source: §1 · p. 1
  • Cross-residual learning extends residual networks with cross-task connections to jointly learn and generalize across related tasks.The proposed multitask network combines shared low-level representations with higher-level representations that specialize and mix information across tasks.Source: §1 · p. 2
  • Adjective-noun pairs can share visual and semantic structure within adjective or noun superclasses and across different superclasses, motivating representations shared across related tasks.For example, “shiny cars” shares adjective-level similarities with “shiny shoes” and noun-level relations with other car concepts.Source: §1 · p. 2
  • The multitask cross-residual network performs competitively or better than specialized networks on noun, adjective, and adjective-noun pair detection, uses >40% less model memory, and outperforms standard multitask residual networks by about 10.4%.The reported 10.4% comparison is against a standard multitask configuration without cross-residuals.Source: §1 · p. 2

2. RELATED WORK

The related work situates the paper among transfer learning, multitask networks, and visual affect research. It distinguishes its approach by continuing to mix task information after branching.

2. RELATED WORK

  • The paper treats related-task learning as a subset of transfer learning, where source and target tasks, domains, or distributions may differ while retaining some relatedness.Relatedness can be semantic or visual, as in classifying dog breeds and bird species or jointly classifying and reconstructing objects.Source: §2 · p. 2
  • Unlike a single branching point that creates task-specific paths, the proposed architecture continues mixing information after branching through cross-skip connections.The related-work discussion contrasts this design with multitask networks that fork to specialize in individual tasks.Source: §2 · p. 3
  • Prior work integrates class relationships within a single-task network, whereas this paper focuses on multitask fan-out and cross-task learning.The cited multimodal network uses multiple input feature towers and a single output head.Source: §2 · p. 3
  • Visual affect research includes adjective-noun pairs as a mid-level semantic construct, and this work uses a deep multitask cross-residual network to predict noun, adjective, and adjective-noun concepts.The cited affect literature also addresses quality assessment, memorability, interestingness, and popularity.Source: §2 · p. 3

3. CROSS RESIDUAL LEARNING

Cross-residual learning extends residual layers to jointly fit related tasks by adding cross-task connections. The resulting layer-level conditioning can be viewed as in-network regularization.

3. CROSS RESIDUAL LEARNING

  • In ordinary residual learning, the layer fits a mapping F(x, {Wi}) and adds it to the input through a shortcut, using a projection when dimensions must match.For identity shortcuts, Ws = I.Source: §3 · p. 3
  • Cross-residual learning adds weighted contributions from other related tasks to a target task’s residual-layer output.The cross-residual contributions may be stacks of operations; the described illustration uses a simple weighted case.Source: §3 · p. 3
  • Cross-residual layers regularize a task earlier in the network by biasing its layer-level residual toward related-task transformations, rather than regularizing only at the loss.The passage compares this in-network regularization to dropout, but notes it has less stochasticity.Source: §3 · p. 3
  • Cross-residual layers can be viewed as ungated highway layers with multiple information paths merging, while cross-residual weights govern cross-task pollination.The paper also describes them as feed-forward ungated LSTMs with additively coupled cell states; LSTM forget gates are analogous to cross-residual weight layers.Source: §3 · p. 4
  • Cross-residual learning resembles ladder networks in combining information at the same structural level, but uses shortcut connections rather than lateral connections.The paper contrasts learned residual perturbations with ladder networks’ injected Gaussian noise and links ladder networks’ semi-supervised setting to less labeled-data requirements.Source: §3 · p. 4

4. MULTITASK CROSS RESIDUAL NETS

The multitask cross-residual network adds cross-task mixing to a branched residual architecture. This design increases late-layer representational power without large input feature spaces, while avoiding the memory-intensive alternative of coupling full network towers.

4. MULTITASK CROSS RESIDUAL NETS

  • Cross-task weights cheaply increase late-layer representational power when branching deep limits each task’s specialization to high-level features.Branching earlier provides larger input feature maps to each task head and can create a large memory footprint.Source: §4 · p. 4
  • Coupling multiple network towers with cross-residual skip connections but no branching creates a memory-intensive composite network feasible only in a multi-GPU environment.Freezing weights, including with greedy layerwise training, could somewhat alleviate this constraint.Source: §4 · p. 4
  • Moving the last ReLU activation and batch normalization outside the residual block creates one of each per task, slightly improving performance in practice.This placement introduces some task specialization by allowing slightly different normalization for each branch.Source: §4 · p. 4

5. MULTITASK VISUAL SENTIMENT

The authors frame visual sentiment detection as three related tasks—adjectives, nouns, and adjective-noun pairs—and evaluate multitask cross-residual networks on a curated VSO subset. The channel-scaled model achieves performance comparable to specialized networks with fewer parameters and improves over unweighted multitask variants.

5. MULTITASK VISUAL SENTIMENT

  • The evaluation treats adjective, noun, and adjective-noun pair detection as related tasks and provides the first VSO benchmarks for noun-only and adjective-only detection.The experiments use a stratified 80/20 split: 307,185 training images and 77,073 test images.Source: §5 · pp. 4–5, §5.2 · p. 6

5.1 Multitask-structured VSO

  • The VSO subset contains 167 nouns, 117 adjectives, 553 adjective-noun pairs, and 384,258 Flickr images, selected to support multitask learning.The source dataset has a long tail; selection criteria retain adjectives paired with at least three nouns, non-overrepresented and nonabstract nouns, and ANPs with at least 500 images.Source: §5.1 · pp. 5–6

5.2 Experiments & Discussion

  • The 50-layer multitask models are compared with four fine-tuned single-task architectures and DeepSentiBank on parameter cost and top-k accuracy.DeepSentiBank was reevaluated on the 553-concept subset without known train/test splits, so its result could be an overestimate.Source: §5.2 · p. 6

5.2.1 Adjective

  • Noun prediction is described as easier than adjective prediction, while ANP difficulty is attributed to visual nuances that scaling may accentuate.The authors contrast those nuanced distinctions with the visual variance involved in adjective recognition.Source: §5.2.1 · pp. 6–7

5.2.2 Effects of Cross-residual Weighting

  • With channelwise cross-residual scaling, the multitask network uses ∼43.2M versus ∼72.4M parameters and improves ANP detection by up to ∼10.37% relative to no weighting.Its detection performance is reported as comparable to the three specialized single-task networks.Source: Table 2 · p. 7, §5.2.2 · p. 7
  • Equal identity cross-task weights reduce accuracy, and removing cross-residuals performs worse than single-task ResNet-50, including ∼9% worse relative ANP detection.Learned cross-task weights are mostly non-negative and are neither uniformly zero nor identity.Source: §5.2.2 · p. 7

5.2.3 Example Multitask Detection Results

  • Example detections show that cross-residual learning can surface intuitively related concepts absent from an image, but can also leave learned relationships insufficiently decoupled.The authors suggest that more computational budget for cross-task weighting layers could model more complicated task relationships.Source: §5.2.3 · p. 8

6. CONCLUSIONS & FUTURE WORK

Cross-residual learning couples related tasks in a multitask network that matched or exceeded specialized networks with over 40% fewer parameters. The paper also reports stronger adjective-noun pair detection than a standard multitask residual network and outlines broader applications and architectural extensions.

6. CONCLUSIONS & FUTURE WORK

  • With >40% fewer parameters, the multitask cross-residual network achieved competitive or better performance than specialized single-task networks and about 10.4% higher relative performance on adjective-noun pair detection than a standard multitask residual network.Adjective-noun pair detection was the hardest of the three related target tasks.Source: §6 · p. 8
  • Figure 6 presents example top-5 classification results for adjective, noun, and adjective-noun pair concepts from the multitask cross-residual network.Source: Figure 6 · p. 8
  • Without cross-residual connections, ANP detection accuracy dropped by ∼9%, while equally weighting cross-task connections bottlenecked information flow.The authors also emphasize the importance of cross-residual weighting over forcing identity cross-residual connections.Source: §6 · p. 8
  • The proposed extensions include applying cross-residual networks to reconstruction, segmentation, domain transfer, and adaptation, and exploring other shortcut transforms and gating.For domain transfer or adaptation, the authors describe learning cross-residual weights while tower weights are frozen.Source: §6 · p. 8

Related papers

Loading 1604.01335v2…