PaperlayerView exact revision on arXiv

Source-linked AI summary

FASA: Feature Augmentation and Sampling Adaptation for Long-Tailed Instance Segmentation

Yuhang Zang, Chen Huang, Chen Change Loy

arXiv:2102.12867v2cs.CV

TL;DR

Long-tailed instance segmentation struggles with rare classes because of scarce training data. FASA adaptively augments and samples virtual features without elaborate loss design, improving performance consistently at low cost and extending to long-tailed classification.

  • Problem

    Long-tailed instance segmentation has severe class imbalance and scarce data for rare classes, causing poor performance on tail categories.

  • Method

    FASA performs online class-wise feature augmentation from observed feature statistics and adapts virtual-feature sampling using validation-loss feedback.

  • Results

    FASA consistently improves instance-segmentation methods, including 9.0% rare-class and 3.3% overall mask AP over Mask R-CNN on LVIS, with ∼3% more training time.

  • Takeaways & Limitations

    FASA is a fast, generic plug-and-play component that works with existing sampling schemes and loss functions and generalizes to long-tailed classification.

  • Takeaways & Limitations

    Some prior feature-augmentation methods face high time or memory costs, and LVIS validation data covers only 871 of 1203 training classes for loss-based adaptation.

Abstract

from arXiv · show

Recent methods for long-tailed instance segmentation still struggle on rare object classes with few training data. We propose a simple yet effective method, Feature Augmentation and Sampling Adaptation (FASA), that addresses the data scarcity issue by augmenting the feature space especially for rare classes. Both the Feature Augmentation (FA) and feature sampling components are adaptive to the actual training status -- FA is informed by the feature mean and variance of observed real samples from past iterations, and we sample the generated virtual features in a loss-adapted manner to avoid over-fitting. FASA does not require any elaborate loss design, and removes the need for inter-class transfer learning that often involves large cost and manually-defined head/tail class groups. We show FASA is a fast, generic method that can be easily plugged into standard or long-tailed segmentation frameworks, with consistent performance gains and little added cost. FASA is also applicable to other tasks like long-tailed classification with state-of-the-art performance.

1. Introduction

Long-tailed instance segmentation suffers from severe rare-class data scarcity, motivating FASA’s adaptive feature augmentation and sampling approach. FASA improves rare and overall performance across settings while adding little training cost.

  • Problem: About 300 LVIS tail classes have no more than 102 positive training instances, producing near-zero predicted probabilities for many rare classes.
  • Motivation: Instance-level re-sampling is preferable to image-level over-sampling because images containing tail objects can also contain additional head-class objects.
  • Method: FASA performs online class-wise feature augmentation using virtual features sampled from distributional priors whose statistics come from previously observed real samples.This lets augmentation capture class distributions while adapting to the evolving feature space.
  • Method: Loss-adapted sampling increases virtual-feature probabilities when validation loss improves and decreases them otherwise, limiting under-fitting or over-fitting.The method adjusts sampling based on each class’s observed performance.
  • Results: 9.0% and 3.3% mask AP improvements over Mask R-CNN for rare and overall LVIS classes, respectively, with only ∼3% more training time.FASA also improves a contemporary loss design by 10.3% and 2.3% on rare and overall classes.
  • Scope: FASA is a plug-and-play component that works with existing sampling schemes or loss functions and generalizes to long-tailed image classification.The paper reports state-of-the-art performance for long-tailed classification.

2. Related Work

Prior long-tailed recognition work uses image- and feature-space augmentation, but instance segmentation imposes distinctive efficiency and modeling constraints. FASA is designed to avoid the costly multi-stage and memory-heavy limitations of several alternatives.

  • Existing approaches: Long-tailed instance-segmentation methods also rely heavily on data re-sampling, while image-space augmentation includes flipping, scaling, rotation, cropping, Mixup, CutMix, and copy-and-paste.
  • Feature-space augmentation: Feature augmentation has benefited face recognition, person re-identification, and long-tailed classification, but remains constrained by instance-segmentation requirements.
  • Limitations of prior work: Existing feature-augmentation methods may require two-phase training or large historical-feature memories, increasing time and memory costs.
  • Limitations of prior work: The instance-segmentation background class lacks a class anchor, making margin-based classification losses less effective for this setting.
  • Limitations of prior work: Small batch sizes in instance-segmentation frameworks limit approaches that rely on mining confusing categories.

3. Methodology

FASA is a plug-and-play module that augments class-wise feature spaces with virtual features and adaptively samples them during training, using online statistics and validation loss.

  • 3. Methodology: FASA combines adaptive Feature Augmentation and adaptive Feature Sampling to generate and selectively use virtual features, especially for rare classes.It is designed as a standalone module that leaves the baseline segmentation framework unaltered.
  • 3. Methodology: FASA is applied to Mask R-CNN’s classification branch, while real and re-sampled virtual features contribute to the classification loss.The classification branch is selected because it is identified as the most vulnerable branch on long-tailed data.
  • 3.1. Adaptive Feature Augmentation: Online class-wise feature means and variances define a Gaussian prior used to generate diverse virtual features through random perturbation.Momentum updates provide more robust estimates from previously observed real features, and diagonal covariance reduces complexity from d^2 to d.
  • 3.3. Experiments: FASA’s ablation is evaluated on LVIS v1.0 using overall, rare, common, and frequent mask mAP to assess its FA and FS components.Table 1 compares the baseline, FA, and adaptive FS configurations.
  • 3.2. Adaptive Feature Sampling: Because validation data may omit classes, FASA clusters training classes into super-groups and adjusts their sampling probabilities using average within-group validation loss.The groups are formed from online feature means and standard deviations with a Fisher’s-ratio distance rather than predefined class-size or semantic heuristics.

4. Experiments

Experiments show that FASA improves rare-class and overall instance-segmentation performance across augmentation comparisons, baselines, training settings, and datasets. Its adaptive sampling changes over training and remains effective when combined with stronger methods and frameworks.

  • 4.1. Ablation Study on LVIS: FASA’s adaptive feature sampling raises rare-class performance from AP_r 12.7% to 17.8%, while the other class groups remain competitive.This ablation isolates the contribution of adaptive feature sampling relative to the baseline and feature augmentation components.
  • 4.1. Ablation Study on LVIS: FASA outperforms or compares favorably with SMOTE, MoEx, InstaBoost, and other feature-augmentation methods on LVIS, with particularly apparent gains for APr and APc.The comparison includes methods designed for interpolation, copy-and-paste augmentation, and long-tailed feature augmentation.
  • 4.2. Comparison with State of the Arts on LVIS: Rare-class sampling probabilities typically increase early, decrease to avoid over-fitting, and bump after learning-rate decay, whereas common and frequent classes change less.Figure 3 also shows changing probabilities within each class group during training.
  • 4.2. Comparison with State of the Arts on LVIS: 9.0% APr and 3.3% AP gains are obtained when FASA is combined with vanilla Mask R-CNN on LVIS, while APc and APf are better or on par.The gains remain consistent across repeated runs and when FASA is combined with RFS, EQL, cRT, BAGS, or Seesaw.
  • 4.2. Comparison with State of the Arts on LVIS: FASA improves Mask R-CNN by 1.9% overall AP and 5.9% APr with ResNet-101, and improves rare-class performance by 4.6% with ResNeXt-101.Benefits also hold for Cascade Mask R-CNN and Seesaw Loss while remaining strong on common and frequent classes.
  • 4.3. Evaluation on COCO-LT: On COCO-LT, FASA improves Mask R-CNN considerably for rare classes and performs better than SimCal without decoupled training.The evaluation compares a Mask R-CNN baseline, SimCal, and Mask R-CNN augmented with FASA.
  • 4.4. Evaluation on CIFAR-LT-100: FASA brings 1.7% accuracy improvement over LDAM, is comparable with M2M, and is more time-efficient than methods requiring classifier pre-training.Its benefits also hold when applied to the stronger De-confound-TDE baseline and exceed Chu et al. with cross-entropy loss.

5. Conclusion

FASA addresses long-tailed instance segmentation by generating virtual features and adapting their sampling, while remaining efficient and broadly applicable. Experiments show gains across settings, methods, and datasets, with especially large improvements for rare classes.

  • 5. Conclusion: FASA generates virtual features for rare categories and uses loss-guided adaptive sampling to avoid over-fitting.It operates as a plug-and-play component with minimal impact on training efficiency.
  • 5. Conclusion: FASA maintains gains across backbones, learning schedules, samplers, and loss functions without compromising common and frequent classes.It also improves other methods including RFS, EQL, cRT, BAGS, and Seesaw.
  • 5. Conclusion: The method generalizes beyond instance segmentation to long-tailed image classification on CIFAR-LT-100.

B. Ablation on Adaptive Feature Sampling

The adaptive feature-sampling ablation compares initialization and adaptation choices, finding inverse-frequency initialization effective and group-wise adaptation more stable. Validation loss is used instead of metrics to reduce evaluation cost.

  • B. Ablation on Adaptive Feature Sampling: Inverse class-frequency initialization achieves better overall mask mAP than uniform initialization and is used by default.Both initialization schemes improve over the no-augmentation baseline; inverse frequency also performs better for rare and frequent classes.
  • B. Ablation on Adaptive Feature Sampling: FASA adapts sampling probabilities to validation loss rather than validation mAP to avoid frequent, expensive metric evaluation.Evaluating validation mAP on the large-scale dataset takes nearly 45 minutes.
  • B. Ablation on Adaptive Feature Sampling: Class grouping is normally used because some classes may be absent from validation, which prevents reliable per-class loss-based adjustment.
  • B. Ablation on Adaptive Feature Sampling: Group-wise sampling adaptation outperforms class-wise adaptation and has lower variance on CIFAR-100-LT.The comparison attributes this to more stable group-wise loss averages than noisy per-class losses.

C. Speed Analysis

FASA adds little computational overhead and is applied to Mask R-CNN’s classification branch because classification error dominates the long-tailed segmentation error profile. Its extra memory is constant in feature dimension.

  • C. Speed Analysis: FASA adds only a small amount of memory and incurs a very small time cost during training.The added memory stores online feature means and variances and depends only on feature dimension.

D. Visualizing Class Grouping Results

FASA forms sampling groups using feature-space clustering, where visually similar, semantically related, and co-occurring classes tend to remain close. Group-wise adaptation is therefore aligned with relationships among classes.

  • D. Visualizing Class Grouping Results: Group-wise adaptation performs better and with lower variance than class-wise adaptation in the reported classification comparison.
  • D. Visualizing Class Grouping Results: Mean-shift clustering forms groups whose visually similar or semantically related classes remain close in feature space.Examples include hairnet with visor and pillow with loveseat.
  • D. Visualizing Class Grouping Results: Co-occurring classes also tend to cluster near one another, supporting joint adjustment of their sampling probabilities.The paper gives pillow and loveseat, and pillow and bedspread, as examples.

E. Evaluation with APFixed and APPool

FASA improves performance under APFixed and APPool, with gains that are especially pronounced for rare categories.

  • 1.1%/1.7% and 1.3%/2.4% gains in overall AP/APr under APFixed for Mask R-CNN and Cascade Mask R-CNN, respectively.The first pair applies to Mask R-CNN and the second to Cascade Mask R-CNN.
  • FASA consistently boosts APFixed and APPool performance, especially for rare categories.APFixed limits detections per class, while APPool evaluates detections across all categories together.
  • For APPool, FASA improves rare-class APr by 2.9% for Mask R-CNN and 2.0% for Cascade Mask R-CNN.Because APPool is class-agnostic, its score is influenced more heavily by frequent classes than rare classes.

F. Implementation Details

The implementation uses standard Mask R-CNN training infrastructure while comparing feature-augmentation methods and documenting their tuning, limitations, and efficiency.

  • Base setup: The implementation uses an ImageNet-pretrained ResNet50 with FPN, Mask R-CNN heads, random horizontal flipping, and multi-scale jittering.The jittered shorter image sizes are 640, 672, 704, 736, 768, and 800.
  • Feature-augmentation baselines: ArcFace-style margin augmentation performs poorly on LVIS because the background class makes it sensitive to the angular-margin hyperparameter m_a.FASA is not limited to a particular loss-function form.
  • Reported implementation results: Tables 13, 14, and 15 report metric definitions, tuned feature-augmentation parameters, and LVIS comparisons with NMS Re-sampling.The reported metrics include overall, rare, common, and frequent mask mAP groups.
  • Transfer limitations: Transferring Chu et al.’s method is limited by small instance-segmentation batch sizes, whereas FASA caches previous-batch features in per-category feature banks.This removes the requirement that confusing categories appear in the same batch.
  • Efficiency: FASA is more efficient than Yin et al. because it uses end-to-end training rather than a two-stage approach.The two-stage alternative introduces additional training-time cost.

G. Comparison with NMS Re-sampling [45]

FASA is compared with NMS Re-sampling on LVIS, where both improve over the baseline and FASA performs slightly better, particularly for rare classes.

  • NMS Re-sampling adjusts thresholds to preserve more rare-class proposals and suppress proposals from frequent classes.Its thresholds for frequent, common, and rare categories are 0.7, 0.8, and 0.9.
  • Figure 6 visualizes predictions for six selected rare classes on the LVIS validation set with and without FASA.The selected classes are saucepan, crouton, date (fruit), koala, softball, and bonnet.
  • FASA performs slightly better than NMS Re-sampling, especially for rare classes, while both outperform the baseline.The comparison is made on the LVIS v1.0 validation set using mask mAP metrics for overall and frequency-based class groups.
  • The reported gap is attributed to NMS Re-sampling changing weights of current samples, whereas FASA also generates new virtual samples.The comparison presents FASA as an orthogonal module that can combine with NMS Re-sampling.

H. Result Visualization

Qualitative results show that FASA improves rare-class predictions in Mask R-CNN on the LVIS validation set.

  • FASA makes rare-class segmentation predictions more accurate, whereas the baseline often assigns small scores or misses the classes.The visualization covers selected rare classes in Figure 6.
Loading 2102.12867v2…