PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal representations:The missing link between faces, text, planktons, and cat breeds

Hakan Bilen, Andrea Vedaldi

arXiv:1701.07275v1cs.CVstat.ML

TL;DR

Machine vision usually relies on separate models for different problems, unlike the broadly reusable representation developed by human vision. The paper tests whether neural networks can learn one representation across diverse domains and finds that extensive sharing works, provided information is carefully normalized.

  • Problem

    Machine vision systems have a narrow scope because different visual problems are typically solved with separate models, motivating the search for universal representations.

  • Method

    The paper jointly learns predictors over multiple visual domains, sharing network components while comparing model capacity with the combined size of the vision problems.

  • Results

    Universal representations appear feasible: extensive parameter sharing works across numerous diverse domains, while normalization—especially domain-specific scaling or instance normalization—supports this sharing.

  • Takeaways & Limitations

    A single neural network can provide a broadly reusable representation across a wide array of real-world visual problems, within the demonstrated scope.

  • Takeaways & Limitations

    The method assumes a similar but disjoint validation set is available for each domain.

Abstract

from arXiv · show

With the advent of large labelled datasets and high-capacity models, the performance of machine vision systems has been improving rapidly. However, the technology has still major limitations, starting from the fact that different vision problems are still solved by different models, trained from scratch or fine-tuned on the target data. The human visual system, in stark contrast, learns a universal representation for vision in the early life of an individual. This representation works well for an enormous variety of vision problems, with little or no change, with the major advantage of requiring little training data to solve any of them. In this paper we investigate whether neural networks may work as universal representations by studying their capacity in relation to the “size” of a large combination of vision problems. We do so by showing that a single neural network can learn simultaneously several very different visual domains (from sketches to planktons and MNIST digits) as well as, or better than, a number of specialized networks. However, we also show that this requires to carefully normalize the information in the network, by using domain-specific scaling factors or, more generically, by using an instance normalization layer.

1. Introduction

The paper asks whether one neural network can serve as a universal representation across highly diverse visual domains, rather than requiring separate models for each problem. It finds that extensive sharing is feasible, but requires careful normalization to address differences in dataset statistics.

  • 1. Introduction: Unlike human vision, machine systems typically use different neural networks for recognition, detection, segmentation, reading text, and species classification.The paper frames this narrow scope as a major limitation despite strong performance on certain tasks.
  • 1. Introduction: The study targets simultaneous learning across extremely diverse domains, going beyond fine-tuning and conventional multi-task settings that share a visual domain.Its setup fixes the task to image classification while combining domains such as text and faces.
  • 1. Introduction: The central question is whether model capacity can keep pace with the combined size of multiple vision problems as their domains become more diverse.If problems overlap, the paper argues that complexity growth can slow enough for universal representations to become possible.
  • 1. Introduction: A single network can share parameters across up to ten diverse visual domains, including CIFAR-10, MNIST, and SVHN, without loss in performance.The authors contrast this capacity with the complexity of combining numerous, diverse vision problems.
  • 1. Introduction: Careful normalization is required for successful sharing, with domain-specific scaling factors compensating for inter-domain statistical shifts.Instance normalization offers a domain-agnostic alternative using one parameter set across domains.

2. Related Work

Related work spans transfer and domain adaptation, knowledge transfer between networks, multi-task learning, and lifelong learning. The paper differs by studying the capacity of one jointly trained representation across multiple tasks and highly diverse domains.

  • Transfer learning and domain adaptation: Transfer and domain-adaptation methods reduce differences between tasks or domains using mechanisms such as matched hidden representations, cross-stitch units, or domain-invariant features.These approaches provide context for sharing information across related tasks or domains.
  • Transfer learning and domain adaptation: Knowledge-distillation methods transfer information from ensembles or larger networks to a single student through prediction or intermediate-representation matching.This line of work transfers information between separately structured networks rather than jointly learning all tasks in one shared model.
  • Multi-task learning: Multi-task learning jointly learns a common representation with task-specific parameters and has been applied to many computer-vision problems.Prior applications include tracking, facial-landmark detection, surface normals, edges, detection, and segmentation.
  • Lifelong learning: Lifelong-learning methods add or learn tasks sequentially while preserving previously acquired knowledge, whereas this paper studies multiple tasks learned jointly.The distinction is explicit in the comparison with Never Ending Learning, Lifelong Learning, and Learning without Forgetting.

3. Method

The method learns predictors across multiple image-classification domains while controlling how much network structure is shared. It compares separate, ordinary shared, and adapted shared representations, using normalization and domain-specific parameters to address dataset differences.

  • The framework learns one predictor per domain while minimizing their average risk, with regularization defining the sharing structure.
  • Separate networks provide the no-sharing baseline, while ordinary feature sharing uses a common subset of layers before domain-specific branches.
  • Adapted feature sharing represents each domain predictor as a shared universal blueprint combined with a small set of domain-dependent parameters.
  • Domain-specific scaling adds a scale and bias after convolutional or linear layers, with a muxer selecting parameters for the current domain.
  • The study evaluates batch and instance normalization, using shared or domain-specific scaling and handling batch statistics differently during testing.
  • Table 2 compares individual training, deep sharing, and full sharing with domain-specific scale and bias across three ten-class datasets.

4. Experiments

Experiments test whether one shared network can handle diverse classification domains, from small datasets to ImageNet and face recognition. They find that extensive sharing is effective, but normalization must account for domain statistics.

  • 4.1. Small datasets: A shared CNN outperforms domain-specific networks on all tasks, reducing average error by 1% while using 10× fewer parameters.The comparison covers the diverse small-dataset experiments.
  • 4.1. Small datasets: Increasing shared-network capacity by 4× and 16× reduces mean error by 0.6% and 1.2% points, respectively, relative to the smaller shared model.Joint training exploits added capacity better than ten independent networks, suggesting overlap among domains.
  • 4.1. Small datasets: Domain-specific batch-normalization moments and scaling achieve the best mean error of 25% across the 10 datasets.Domain-agnostic scaling with domain-specific moments reaches 27.3%, while BN with domain-agnostic moments and scaling performs near chance.
  • 4.1. Small datasets: Instance normalization works with either domain-specific or domain-agnostic scaling, trading a 5% performance drop for one parameter set across domains.The authors suggest this may support novel domains without domain-specific normalization tuning.
  • 4.2. Large datasets: On ImageNet and VGG-Face, sharing convolutional weights yields comparable performance, with only a marginal 1% accuracy drop.The experiment uses AlexNet trained jointly on the two large datasets.
  • 4.2. Large datasets: A jointly trained ImageNet, VGG-Face, and Synth90k model performs dramatically better than equal-parameter independent models, despite a 26.9% Synth90k error rate.The joint model uses one third of the parameter count of the corresponding individual networks.

5. Conclusions

Universal representations can serve many visual domains, but effective sharing requires normalization that compensates for differences between datasets.

  • Universal representations are feature extractors intended to work across diverse visual domains, bringing machine vision closer to human-like breadth.
  • Standard deep neural networks can learn highly diverse visual domains together while sharing substantial information.
  • Successful sharing requires tuning normalization parameters, preferably with domain-specific scaling factors that compensate for inter-domain statistical shifts.
  • Instance normalization offers a domain-agnostic alternative that compensates for statistical differences without domain-specific normalization parameters.
  • The findings suggest universal representations are within reach for a wide array of real-world problems, although the strongest results involve smaller datasets.
Loading 1701.07275v1…