PaperlayerView exact revision on arXiv

Source-linked AI summary

Factors in Finetuning Deep Model for object detection

Wanli Ouyang, Xiaogang Wang, Cong Zhang, Xiaokang Yang

arXiv:1601.05150v2cs.CV

TL;DR

The paper studies how class imbalance and class-specific visual structure influence finetuning for object detection. It analyzes these factors and proposes cascaded hierarchical feature learning that transfers knowledge from generic groups to specific visually similar groups. The approach achieves a 4.7% absolute mAP improvement on ImageNet object detection without much additional testing cost.

  • Problem

    The paper investigates how long-tailed class frequencies and shared feature representations affect finetuning effectiveness in object detection.

  • Method

    The paper groups visually similar object classes and uses hierarchical finetuning to transfer knowledge from larger generic groups to smaller specific groups through a cascade.

  • Results

    4.7% absolute mAP improvement was achieved on the ImageNet object detection dataset without much increase in testing computational cost.

  • Takeaways & Limitations

    Feature learning benefits from more uniform class sample numbers and from representations focused on specific object-class groups.

  • Takeaways & Limitations

    The experiments change one factor at a time while keeping the others fixed, and the comparison uses a specified 40.3% baseline for consistency.

Abstract

from arXiv · show

Finetuning from a pretrained deep model is found to yield state-of-the-art performance for many vision tasks. This paper investigates many factors that influence the performance in finetuning for object detection. There is a long-tailed distribution of sample numbers for classes in object detection. Our analysis and empirical results show that classes with more samples have higher impact on the feature learning. And it is better to make the sample number more uniform across classes. Generic object detection can be considered as multiple equally important tasks. Detection of each class is a task. These classes/tasks have their individuality in discriminative visual appearance representation. Taking this individuality into account, we cluster objects into visually similar class groups and learn deep representations for these groups separately. A hierarchical feature learning scheme is proposed. In this scheme, the knowledge from the group with large number of classes is transferred for learning features in its sub-groups. Finetuned on the GoogLeNet model, experimental results show 4.7% absolute mAP improvement of our approach on the ImageNet object detection dataset without increasing much computational cost at the testing stage.

1. Introduction

The paper examines how long-tailed class frequencies and shared feature representations affect finetuning for object detection. It proposes cascaded hierarchical feature learning, transferring knowledge from generic class groups to visually specific subgroups.

  • 1. Introduction: Finetuning initializes a target-task model from parameters pretrained on a related task and has achieved strong results across vision tasks.The cited applications include tracking, segmentation, object detection, action recognition, and human pose estimation.
  • 1. Introduction: Object-detection datasets exhibit long-tailed class frequencies, with a few classes appearing often and most classes appearing rarely.Person has many more samples than sheep in both PASCAL VOC and ImageNet detection datasets.
  • 1. Introduction: Around 40% fewer positive samples still slightly improved detection accuracy when samples were made more uniform across classes.This finding motivates examining sample-number imbalance during feature learning.
  • 1. Introduction: Because each object class forms an independently evaluated detection task, the paper questions whether one shared representation is optimal for all classes.The paper argues that visually specific representations can better describe class-specific appearance.
  • 1. Introduction: The paper investigates finetuning factors including layer adaptation, long-tail effects, training sample number, class subsets, and training-data subsets.These experiments provide the paper’s analysis of what influences finetuning effectiveness.
  • 1. Introduction: 4.7% absolute mAP increase was achieved on ImageNet object detection using cascaded hierarchical feature learning.Object classes are grouped, models progressively focus on specific groups, and knowledge transfers from larger generic groups to smaller specific groups.

2. Related work

Related work addresses long-tail learning, deep-learning applications, object-detection systems, and cascaded or ensemble models. The paper distinguishes its approach by learning different representations for different object-class groups.

  • 2. Related work: Prior long-tail research expanded rare-class samples for more balanced superpixel classification and used semantic similarity to predict unseen classes.The paper notes that deep learning offers representation sharing, but says the long-tail influence on deep feature learning had not been investigated to its knowledge.
  • 2. Related work: Deep learning has been applied broadly across many vision tasks and extensively to object detection.Existing object-detection work mainly develops new deep models or improves detection pipelines.
  • 2. Related work: Existing approaches generally use one feature representation for all object classes, whereas this work uses different representations for different class groups.The supplied passage introduces this contrast but truncates the detailed description of the proposed representations.
  • 2. Related work: Unlike model ensembles, the proposed approach obtains each object class’s detection score from only one model and is described as complementary to ensembling.Model ensembles combine scores from multiple models or architectures.
  • 2. Related work: Cascade mechanisms have appeared in prior object-detection work, while this paper uses cascade to speed up testing.

3. Factors in finetuning for ImageNet object detection

The study finds that finetuning performance depends strongly on training-data composition, layer choice, and class balance. Uniformizing class samples improves feature learning, while upper-layer finetuning and broad class coverage support higher detection mAP.

  • Layer finetuning: Finetuning upper GoogLeNet layers has greater impact on detection performance because higher layers are more responsible for semantic object discrimination.Freezing progressively higher-level modules causes mAP to decrease more rapidly.
  • Long-tail property: 59.5% of ground-truth samples come from the 20 object classes with the largest sample counts, demonstrating a strong long-tail distribution.The long tail persists even after constraining train13 samples to at most 1,000 per class.
  • Long-tail property: Classes with more samples can dominate shared feature learning, potentially producing weaker features for rare classes despite equal class importance in detection.The paper contrasts bird, with many samples, against hamster, with few samples.
  • Long-tail experiments: Pseudo-uniform sampling achieves mAP 39.9% at r = 12.5% using half as many positive samples as rand-pos at r = 25%.The baseline using all samples has 40.3% mAP, while retaining roughly 40% fewer positive samples with Nmax = 3,000 yields 40.5% mAP.
  • Long-tail experiments: 40.7% mAP results when all training samples are retained but mini-batches enforce a uniform positive-sample distribution across classes.These results support uniform class representation over long-tailed sampling for feature learning.
  • Class subsets: Using 150 high-sample classes for finetuning reaches 40.1% mAP, compared with 40.3% when all 200 classes are used.Adding the 50 lowest-sample classes increases mAP by only 0.2%.

4. Cascaded hierarchical feature learning for object detection

The approach groups object classes into hierarchical clusters using visual similarity and finetunes multiple models for those groups. At testing, samples are evaluated from root to leaves, with child groups evaluated only when parent scores do not reject them.

  • Hierarchical clustering: Object classes are grouped into hierarchical clusters, with each child group contained within its parent and parent groups formed from their children.The hierarchy contains nested class sets S_l,j_l and corresponding models M_l,j_l.
  • Hierarchical clustering: Visual similarity between classes is computed from inner products of their last GoogLeNet hidden-layer representations.The similarity uses hidden representations from training samples of classes a and b.
  • Hierarchical clustering: The hierarchy uses four levels, with 1, 4, 7, and 18 groups and average group sizes of 200, 50, 29, and 11 classes.The initial root contains all 200 ImageNet object-detection classes.
  • Cascaded testing: At testing, samples proceed from the root toward leaves, and parent detection scores determine which child groups require evaluation.Only groups that are not rejected retain final class-level SVM scores.
  • Cascaded testing: The testing procedure takes a sample and hierarchical clusters as input and outputs a detection-score vector across object classes.The algorithm uses models associated with the hierarchy during evaluation.

12 end

Hierarchical learning assigns one finetuned model to each class group, using parent models as initialization and restricting training samples to the current group and parent-accepted negatives.

  • Hierarchical learning: Each hierarchy node corresponds to a class group S_l,j_l and a finetuned deep model M_l,j_l.The algorithm outputs the finetuned models associated with the hierarchical clusters.
  • Hierarchical learning: Positive samples for a node are restricted to labels within its class group, while negative samples must be accepted by the parent node.This restriction makes the model focus on representations for the node’s class subset.
  • Hierarchical learning: A node model is initialized from its parent model so knowledge transfers from broader groups to more specific groups.The root model is pretrained for the 1000-class problem before hierarchical finetuning proceeds.
  • Hierarchical learning: The algorithm uses node models to compute detection scores for child-group classes on selected negative samples.For each child group, the score is the maximum class score within that group.

11 end

The cascade combines inherited representations with progressively more specific finetuning. It also focuses later models on hard negatives and uses separate losses for feature learning and classification.

  • Hierarchical feature learning: Models at deeper hierarchy levels inherit knowledge from both the 1000-class pretrained model and the 200+1-class finetuned model.Parent initialization carries knowledge through the hierarchy.
  • Hierarchical feature learning: Cascade-based negative sampling makes each child model focus on hard examples that its parent model handles poorly.The child model is trained on negatives accepted by the parent node.
  • Training losses: Feature representations are learned with multiclass cross-entropy, followed by a two-class hinge loss for the classifier.The classifier is trained after the deep feature representation is learned.

5. Experimental results on the Hierarchical Feature Learning

The experiments show that visually informed class grouping and deeper hierarchical specialization improve detection accuracy, while cascading keeps evaluation cost manageable. Gradual finetuning through intermediate class groups performs better than abrupt specialization.

  • 5.2.1 Investigation on different clustering methods: All clustering approaches improved detection accuracy except random assignment; confusion-matrix and visual-similarity clustering performed best.The 200 classes were divided into four groups from a 40.3% mAP baseline.
  • 5.2.1 Investigation on different clustering methods: Visual similarity and confusion-matrix clustering produced similar results because visually similar objects often cause detection confusion.Visual similarity was adopted for the final implementation because it performed better than the other approaches empirically.
  • 5.2.2 Investigation on the influence of hierarchy level: 45% mAP was achieved at hierarchy level 4, increasing consistently from 40.3% at level 1.Higher levels learn more specific representations that better distinguish objects from background.
  • 5.2.2 Investigation on the influence of hierarchy level: The cascade evaluates only about 5.6 boxes per image per model despite using 18 models at level 4.Cascaded rejection substantially reduces the apparent evaluation burden of deeper hierarchies.
  • 5.2.3 Investigation on finetuning strategies: 42.5% mAP resulted from finetuning level-3 models from level 2, versus 41.8% from level 1.Gradually focusing from 200 to 50 and then 29 classes outperformed the direct jump from 200 to 29 classes.
  • 5.2.4 Results on the PASCAL VOC: 1.2% mAP improvement was observed on PASCAL VOC 2007 when object classes were clustered into four groups.This extends the clustering result beyond the ImageNet detection experiments.

6. Conclusion

The paper identifies sample imbalance as a factor in object-detection finetuning and proposes cascaded hierarchical feature learning. The approach improves learned features and achieves a 4.7% absolute mAP gain without much additional testing cost.

  • 6. Conclusion: Uniform sample numbers across classes are better for feature learning in object detection.The conclusion summarizes the paper’s analysis of the long-tailed class distribution.
  • 6. Conclusion: Cascaded hierarchical feature learning improves the effectiveness of learned features.The scheme specializes representations through class-group hierarchies.
  • 6. Conclusion: 4.7% absolute mAP improvement was achieved without much increase in computational cost.This is the paper’s reported overall outcome for the proposed scheme.
Loading 1601.05150v2…