PaperlayerView exact revision on arXiv

Source-linked AI summary

Network of Experts for Large-Scale Image Categorization

Karim Ahmed, Mohammad Haris Baig, Lorenzo Torresani

arXiv:1604.06119v3cs.CV

TL;DR

Large-scale image categorization must distinguish many classes, including fine distinctions that can require specialist expertise. The paper learns a generalist and class specialties for a tree-structured network of experts, then fine-tunes the unified model. Across adapted CNNs, it reports improved accuracy at little additional training cost.

  • Problem

    Large-scale image categorization must distinguish many classes, including fine distinctions that can require domain-specific expertise.

    Source: §1 · p. 1

  • Method

    The method jointly learns class specialties and a generalist, then uses specialty-specific expert branches in a unified model fine-tuned over the original classes.

    Source: §1 · p. 2

  • Results

    Across adapted CNNs, NofE improves accuracy at little added training cost; on CIFAR100, accuracy is 56.2% versus 54.0%.

    Source: Number of Experts and Specialty Balance · p. 10, §5 · p. 14

  • Takeaways & Limitations

    The approach improves accuracy across the tested CNN adaptations while keeping additional training cost small.

    Source: §5 · p. 14

    The approach is not assessed on single-domain fine-grained benchmarks, which the authors say do not require a generalist and multiple specialists.

    Source: §1 · p. 3

Abstract

from arXiv · show

We present a tree-structured network architecture for large scale image classification. The trunk of the network contains convolutional layers optimized over all classes. At a given depth, the trunk splits into separate branches, each dedicated to discriminate a different subset of classes. Each branch acts as an expert classifying a set of categories that are difficult to tell apart, while the trunk provides common knowledge to all experts in the form of shared features. The training of our "network of experts" is completely end-to-end: the partition of categories into disjoint subsets is learned simultaneously with the parameters of the network trunk and the experts are trained jointly by minimizing a single learning objective over all classes. The proposed structure can be built from any existing convolutional neural network (CNN). We demonstrate its generality by adapting 4 popular CNNs for image categorization into the form of networks of experts. Our experiments on CIFAR100 and ImageNet show that in every case our method yields a substantial improvement in accuracy over the base CNN, and gives the best result achieved so far on CIFAR100. Finally, the improvement in accuracy comes at little additional cost: compared to the base network, the training time is only moderately increased and the number of parameters is comparable or in some cases even lower.

1 Introduction

The paper frames large-scale categorization as a generalist–specialist problem: broad categories are recognizable, but fine distinctions can require domain-specific expertise. It proposes jointly learning class specialties and a unified tree-structured network of experts, while using generalist features to initialize the shared trunk; the approach is scoped to large-class problems rather than single-domain fine-grained categorization.

1 Introduction

  • Recognizing fine distinctions across many categories can require dedicated expertise, unlike recognizing broad visual categories.The introduction illustrates this contrast with identifying mushrooms, authenticating art, and diagnosing medical images.Source: §1 · p. 1
  • The generalist partitions classes into learned specialties, while expert branches classify categories within each specialty in a unified, end-to-end network.The generalist’s convolutional layers initialize the shared trunk, and the complete model is fine-tuned for the original classes.Source: Figure 1 · p. 2, §1 · pp. 2–3
  • The specialties group confusable classes so experts can focus on difficult distinctions, while generalist-trained convolutional layers provide shared features that improve fine-tuning.The authors report better fine-tuning than training the network of experts from scratch or using convolutional layers optimized over all classes.Source: §1 · p. 3
  • The approach is tested on large-class categorization; single-domain fine-grained benchmarks are not assessed because they do not require a generalist and multiple specialists.The authors cite ImageNet-style coarse and confusable categories as matching the model’s structure.Source: §1 · p. 3

2 Related Work

The paper relates its approach to CNN architecture changes, learned category hierarchies, and prior systems that train expert networks. It distinguishes its method through direct specialty learning, disjoint experts evaluated in parallel, and fine-tuning of the complete model on the original classification objective.

2 Related Work

  • The work also connects to CNN structural changes and category hierarchies used for faster classification, abstraction, specificity–accuracy trade-offs, and rare-class transfer.It characterizes its own gains as coming from adapting existing CNNs into a tree structure.Source: §2 · pp. 3–4
  • Related expert systems include ensemble specialists, auxiliary specialist pathways, and a hierarchical coarse-and-fine CNN with late fusion.The paper says these systems are similar in spirit but differ substantially in architecture.Source: §2 · p. 4
  • Unlike prior expert systems, this method learns a generalist over K specialties rather than first training a large CNN over all C classes.The authors state this avoids the expensive base-model training stage and can keep the parameter count below that of the base model.Source: §2 · p. 4
  • Unlike routing systems, its disjoint specialties invoke all experts in parallel, so a routing mistake cannot exclude an expert’s prediction.The cited systems use overlapping specialties to reduce routing errors, increasing the classes each expert must recognize.Source: §2 · p. 4
  • Unlike the related expert systems, the complete network is fine-tuned with one objective over the original C categories.The paper says this fine-tuning was not done in practice for and because of computational cost and parameter count.Source: §2 · p. 4

3 Technical approach

The method learns class specialties and adapts a base CNN into a tree-structured network of experts. It trains a generalist first, then jointly trains the experts for classification over the original classes.

3 Technical approach

  • The method adapts a base CNN architecture and learning objective in two stages: learn a generalist, then train the complete network of experts.The base architecture's weights need not be pre-optimized; the generalist initializes the later network's trunk.Source: §3 · p. 5

3.1 Learning the Generalist

  • Specialties are K << C disjoint class subsets, learned by alternating CNN-weight optimization with updates to the class-to-specialty mapping.The first stage optimizes recognition of specialty labels rather than the original class labels.Source: §3.1 · pp. 5–6
  • Confusion-based label updates yielded highly imbalanced specialties, so the authors use fully balanced assignments or elasso to constrain or softly encourage equal sizes.The fully balanced scheme forces equal specialty sizes, while elasso softly encourages the size constraint.Source: §3.1 · pp. 6–7

3.2 Training the Network of Experts

  • The final network uses the generalist's convolutional layers as a shared trunk, with one expert branch per specialty and a global softmax over all C classes.Backpropagation updates all K branches for each example, jointly calibrating expert outputs over the original classes.Source: §3.2 · pp. 7–8

4 Experiments

Experiments evaluate Networks of Experts (NofE) on CIFAR100 and ImageNet, using CIFAR100 for broader design studies and ImageNet for large-scale assessment. Across the tested base CNNs, NofE improves accuracy.

Varying the Base CNN.

Number of Experts and Specialty Balance.

Defining Specialties with Other Schemes.

Varying the Base CNN.

Finetuning NofE from generalist vs. learning from scratch

  • Pretraining the NofE trunk with the generalist improves CIFAR100 accuracy over training from scratch: AlexNet-C100 rises from 49.53% to 56.2%, and ResNet56-C100 from 73.95% to 76.24%.Source: Finetuning NofE from generalist vs.… · p. 12

4.2 Categorization on ImageNet

  • On ImageNet, the learned-specialty generalist eventually matches the spectral generalist, but the resulting spectral NofE falls below the base model; WordNet and random specialties perform worse.Source: §4.2 · p. 13, Figure 2 · p. 14
  • NofE adds training time: on CIFAR100, its training-time ratio versus the base model is about 1.5×, while ImageNet training takes about 12 days versus 7 days for the base model.Source: §4.2 · pp. 13–14

5 Conclusions

The paper introduces a tree-structured model that learns coarse specialties and specialized experts jointly, then fine-tunes the unified model for the original classes. Across adapted CNNs, it reports improved accuracy with little added training cost.

5 Conclusions

  • The Network of Experts combines a generalist that distinguishes coarse specialties with expert networks dedicated to classes within each specialty.The specialties and generalist are learned jointly, and the resulting tree-structured model is fine-tuned to recognize the original classes.Source: §5 · p. 14
  • Adapting several popular CNNs into networks of experts improved accuracy in each case, with little added training cost.Source: §5 · p. 14
  • The paper reports that its approach learns the specialties and generalist jointly, then fine-tunes a single model for the original classification objective.Source: §5 · p. 14

A.1 Nearest Neighbor Retrieval

The supplementary retrieval experiments compare expert features with base-model features on CIFAR100 and ImageNet. The reported examples show class-matching CIFAR100 neighbors more often with expert features, and semantically related ImageNet neighbors.

A.1 Nearest Neighbor Retrieval

  • On CIFAR100, expert features retrieved many neighbors matching the query class, while base-model features produced many mismatches.The comparison used AlexNet-C100 and the last convolutional layer of the expert branch corresponding to the query's predicted class.Source: §A.1 · p. 15
  • The two retrieval comparisons use expert-branch features against base-model features, with the feature layer differing between CIFAR100 and ImageNet.CIFAR100 uses the last convolutional layer; ImageNet uses the first fully connected layer.Source: §A.1 · p. 15
  • On ImageNet, expert-feature retrieval included many same-class images or, when mistaken, images of related classes such as a lynx for a cougar query.The expert features came from the first fully connected layer of the branch associated with the query's predicted class.Source: §A.1 · p. 15

A.2 Learned Specialties

The learned specialties group many semantically or visually similar categories, though outliers occur. Increasing CIFAR100 specialties from K=10 to K=20 made the groups more homogeneous but did not improve final accuracy over K=10.

A.2 Learned Specialties

  • At K=10 on CIFAR100, most learned specialties contained semantically or visually similar classes, but some outliers remained.The example given is “skyscraper” in a specialty otherwise containing natural classes.Source: §A.2 · p. 15, Table 6 · p. 18
  • With K=20, CIFAR100 specialties were more homogeneous than with K=10, but final categorization accuracy did not improve over K=10.The authors suggest the higher parameter count at K=20 may cause overfitting.Source: §A.2 · pp. 15, 18, Table 7 · p. 18
  • The ImageNet examples show specialties learned with AlexNet and K=10.Source: §A.2 · p. 18, Table 8 · p. 18

A.3 Finetuning from the base model

The authors compare training the Network of Experts from scratch with initializing its generalist from a pretrained base model. The scratch-trained setup performed better in their AlexNet-C100 experiment.

A.3 Finetuning from the base model

  • The authors report that NofE training from scratch does not require first learning the base model, making the procedure faster and more streamlined.Source: §A.3 · p. 18
  • 56.2% accuracy from scratch exceeded the 55.5% achieved by finetuning the generalist from the pretrained AlexNet-C100 base model.Source: §A.3 · p. 19
  • The authors suggest base-model features provide a poor initialization for the generalist and subsequent expert branches in this experiment.The compared setup finetuned the generalist from AlexNet-C100 before training the full NofE.Source: §A.3 · p. 19

A.4 Depth vs Specialization

An ImageNet control with one extra convolutional layer scored below both the shallower base network and the Network of Experts (NofE). The authors attribute the NofE’s gain to expert specialization rather than added depth.

A.4 Depth vs Specialization

  • 61.29% for NofE exceeded 56.91% for the depth-matched AlexNet-Caffe variant, which scored below the shallower base network.The NofE result is reported as much higher than the deeper variant.Source: §A.4 · p. 19
  • The authors conclude that specialization by the experts, rather than the additional convolutional layer, drives the improvement.The NofE typically has one more convolutional layer than its base model.Source: §A.4 · p. 19

A.5 Distribution of specialty sizes

The authors report that specialty-size balance matters and use elasso’s λ to trade off balanced specialties against categorization accuracy. Small λ values yield uneven specialty sizes, whereas large values produce nearly balanced groups.

A.5 Distribution of specialty sizes

  • Elasso’s λ trades off balanced specialty sizes against categorization accuracy over the specialties.The authors report that balancing specialty sizes is important.Source: §A.5 · p. 19
  • λ values of 100 or 200 produce highly uneven specialty sizes for K = 10 experts.Source: §A.5 · p. 19
  • Large λ values yield perfectly balanced specialty sizes for K = 10 experts.Source: §A.5 · p. 19

A.6 Network Specifications

The appendix specifies the CNNs used for CIFAR100 and ImageNet and explains the notation used for their Network of Experts variants. The specifications describe shared generalist/base architectures, expert branches, and a fully balanced specialty assumption.

A.6 Network Specifications

  • The listed architectures cover AlexNet-C100, AlexNet-Quick-C100, VGG11-C100, NIN-C100, ResNet56-C100, and AlexNet-Caffe for ImageNet.Tables 9–13 specify the CIFAR100 models and Table 14 specifies the ImageNet model.Source: §A.6 · pp. 19–20, Table 9 · p. 22, Table 10 · p. 23
  • Figure 5 sorts specialty sizes for K = 10 across λ values 100, 200, 500, 1000, and 10000; small λ gives uneven groups, while large λ makes them nearly equal.Source: Figure 5 · p. 20
  • Each specification contrasts a generalist/base model pair sharing an architecture, except for the final fully connected layer’s units and training policy, with a NofE formed by replacing that layer with K expert branches.Source: §A.6 · p. 20

Notation:

  • Layer notation covers convolution, pooling, local response normalization, and fully connected layers; all architectures use ReLUs.The notation gives examples for convolution and pooling layers and defines FC output size.Source: §A.6 · p. 20, Notation · p. 20
  • The tables assume fully balanced specialties with C/K classes each, and use braces for residual blocks and “Expert(i) →” for branch stacks connected in parallel to the trunk.The residual-block example is a concatenation of nine two-layer convolutional blocks.Source: Notation · p. 21

A.7 Software Implementation

The implementation is based on Caffe, which the authors modified with new layers, solvers, and tools to support the Network of Experts. They also report making software and pretrained models available.

A.7 Software Implementation

  • The Network of Experts implementation is based on Caffe [20], with modifications including new layers, solvers, and tools.Source: §A.7 · p. 28
  • The authors report that software implementing the method and several pretrained NofE models are available.Source: §A.7 · p. 28
Loading 1604.06119v3…