PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Representation Learning for Natural World Image Collections

Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, Oisin Mac Aodha

arXiv:2103.16483v2cs.CV

TL;DR

Fine-grained natural-world vision offers abundant images but costly labeling, while most self-supervised research has focused on ImageNet. The paper introduces iNat2021 and NeWT and evaluates supervised and self-supervised representations across natural-world tasks, finding that supervised features generally remain stronger. These datasets provide benchmarks for tracking progress in natural-world representation and transfer learning.

  • Problem

    Natural-world image collections contain information beyond species labels, but specialized fine-grained tasks require costly labeling and self-supervised methods have mostly been studied on ImageNet.

  • Method

    The paper introduces iNat2021 and NeWT, then evaluates ResNet50 representations pretrained on ImageNet or iNaturalist data with supervised and self-supervised objectives across downstream tasks.

  • Results

    Supervised features generally outperform existing self-supervised features across the tested natural-world tasks, although SwAV closes the gap and performs better in some cases.

  • Takeaways & Limitations

    iNat2021 and NeWT provide resources for evaluating representation learning and transfer on fine-grained natural-world categories and broader visual-understanding tasks.

  • Takeaways & Limitations

    The supervised–self-supervised performance gap is much larger after iNat2021 pretraining, potentially indicating that current self-supervised settings are overfit to ImageNet.

Abstract

from arXiv · show

Recent progress in self-supervised learning has resulted in models that are capable of extracting rich representations from image collections without requiring any explicit label supervision. However, to date the vast majority of these approaches have restricted themselves to training on standard benchmark datasets such as ImageNet. We argue that fine-grained visual categorization problems, such as plant and animal species classification, provide an informative testbed for self-supervised learning. In order to facilitate progress in this area we present two new natural world visual classification datasets, iNat2021 and NeWT. The former consists of 2.7M images from 10k different species uploaded by users of the citizen science application iNaturalist. We designed the latter, NeWT, in collaboration with domain experts with the aim of benchmarking the performance of representation learning algorithms on a suite of challenging natural world binary classification tasks that go beyond standard species classification. These two new datasets allow us to explore questions related to large-scale representation and transfer learning in the context of fine-grained categories. We provide a comprehensive analysis of feature extractors trained with and without supervision on ImageNet and iNat2021, shedding light on the strengths and weaknesses of different learned features across a diverse set of tasks. We find that features produced by standard supervised methods still outperform those produced by self-supervised approaches such as SimCLR. However, improved self-supervised learning methods are constantly being released and the iNat2021 and NeWT datasets are a valuable resource for tracking their progress.

1. Introduction

Self-supervised learning could reduce the costly labeling burden in fine-grained natural-world vision, where image collections contain information beyond species identity. The paper introduces iNat2021 and NeWT to study representation learning and transfer across such tasks.

  • Motivation: Fine-grained domains often have abundant images but costly, time-consuming labeling because qualified annotators are scarce or expensive.This motivates representation learning methods that can exploit large unlabeled media collections.
  • Motivation: Natural-world repositories contain evidence about individual health, state, behavior, and ecology beyond species labels.Automated analysis could help fill gaps in natural-history knowledge and measure biodiversity loss due to human impact.
  • Problem: Specialized ecological questions remain difficult to support with labeled datasets large enough for off-the-shelf deep networks.The paper illustrates this boundary with poorly understood bird nesting requirements and conservation-related questions.
  • Approach: The study expands self-supervised learning from predominantly ImageNet-based research to natural-world fine-grained classification and evaluates whether representations generalize across tasks with limited supervision.The paper treats broad downstream generalization as a criterion for representation quality.
  • Contributions: The paper introduces iNat2021, containing over 2.7M images from 10k species, and NeWT, a suite of 164 natural-world visual benchmark tasks.NeWT targets real-world image-understanding questions beyond standard species classification, while the paper evaluates self-supervised representation learning in this domain.

2. Related Work

Prior work established supervised transfer learning, self-supervised representation learning, and fine-grained datasets, but evaluation remained largely centered on standard datasets and species classification. This paper compares representation-learning objectives on natural-world datasets and introduces broader, expert-informed evaluation tasks.

  • Learning Visual Representations: Supervised pretraining on large datasets produces features that transfer to many downstream tasks, but ImageNet can be less effective for fine-grained categories absent from its labels.This motivates studying alternative pretraining data and objectives for fine-grained transfer.
  • Learning Visual Representations: Recent self-supervised methods learn representations by bringing augmented views of the same image closer while separating other images in feature space.Contrastive objectives are described as the dominant effective recent approach.
  • Benchmarking Representation Learning: The paper compares leading self-supervised algorithms with conventional supervised learning for fine-grained natural-world classification, decoupling feature learning from classification.The evaluation focuses on the gap between representations learned by the two paradigms.
  • Benchmarking Representation Learning: Most self-supervised research trains on ImageNet, whereas this work learns representations through self-supervision on a large-scale natural-world dataset rather than only linear evaluation or domain finetuning.This places self-supervised pretraining itself in the natural-world setting.
  • Fine-Grained Datasets: NeWT broadens fine-grained evaluation beyond species classification to expert-informed behavior, health, and context tasks grounded in real-world use cases.Paired with iNat2021, it supports evaluation of transfer and self-supervised learning in challenging visual domains.

3. The iNaturalist 2021 Dataset

iNat2021 is a large-scale, fine-grained natural-world dataset with 2.7M training images across 10k species, plus a smaller mini split and metadata for transfer-learning research. Baseline experiments show substantially higher supervised accuracy with the full dataset, while random initialization approaches ImageNet initialization at full scale.

  • iNat2021 contains 2.7M training images, 100k validation images, and 500k test images spanning 10k species across the tree of life.
  • Each species has at least 152 training images, with train and test images split by submission date.
  • iNat2021 mini provides 50 training images per species, enabling comparisons between desktop-scale training and five times more full-dataset data.
  • The full dataset is more than five times larger than existing large-scale species-classification datasets, while both iNat2021 variants share validation and test sets.
  • 65.4% top-1 accuracy on the mini training split increased to 76.0% on the full split as training images rose from 500k to 2.7M.
  • Randomly initialized models perform slightly worse than ImageNet-initialized models, but the gap closes when training uses the full dataset.

4. NeWT: Natural World Tasks

NeWT is a curated benchmark of 164 binary natural-world image tasks designed around real-world questions beyond species identification. Its tasks span organism appearance, context, counting, and photograph gestalt, with expert-assisted quality review.

  • Unlike species-only benchmarks, NeWT includes tasks concerning behavior, health, context, and other natural-world image properties.
  • NeWT contains 164 curated binary classification tasks sourced from iNaturalist, the Macaulay Library, NABirds, and other collections.
  • Domain experts vetted each NeWT task for data quality, and no NeWT images occur in the iNat2021 training set.
  • Appearance: Appearance tasks cover age, attributes, health, and species, totaling 14, 7, 9, and 102 tasks respectively.

5. Experiments

The experiments evaluate supervised and self-supervised representations across existing fine-grained datasets and NeWT tasks using a common feature-extraction and linear-classification protocol. Supervised representations generally perform best, while self-supervised performance depends on the dataset and task type.

  • Experimental setup: The evaluation uses ResNet50 representations pretrained on ImageNet or iNaturalist datasets, then trains linear classifiers on downstream features.Features come from the last convolutional block, with dimension 2048 or 8192 for 4x-width models.
  • Fine-grained datasets: Supervised methods produce the best features on every existing fine-grained dataset except Stanford Cars, where ImageNet SwAV performs best.The supervised iNat2021 model leads on Flowers102, CUB, and CUBExpert; iNat2018 narrowly leads iNat2021 on NABirds at 0.806 versus 0.804 top-1 accuracy.
  • Fine-grained datasets: ImageNet generally outperforms iNat2021 as the pretraining dataset for self-supervised methods on existing fine-grained datasets, except Flowers102.This pattern also appears on bird tasks despite iNat2021 containing more species and training images.
  • NeWT: 0.806 average accuracy is achieved by supervised iNat2021 across the 164 NeWT tasks, compared with 0.744 for supervised ImageNet and 0.733 for the best self-supervised model.The best self-supervised result is ImageNet-trained SwAV.
  • NeWT: NeWT results vary by task: supervised iNaturalist pretraining benefits Appearance and Behavior tasks, whereas self-supervised features outperform supervised methods on Gestalt tasks.No method significantly improves Context tasks, and Counting tasks remain difficult for all methods.
  • Discussion: The gap between supervised and self-supervised features is larger for iNat2021-trained extractors than for ImageNet-trained extractors, suggesting ImageNet-specific training settings may not generalize.The authors identify hyperparameters or image augmentations as possible sources of this mismatch.

6. Conclusion

The paper introduces and benchmarks iNat2021 and NeWT for natural-world representation learning. Supervised learning still outperforms existing self-supervised approaches, but the datasets provide challenging benchmarks for future methods.

  • iNat2021 contains 2.7M training images covering 10k species and supports pretraining for fine-grained datasets and NeWT.
  • NeWT extends wildlife image understanding beyond species identification to behavior, health, and contextual questions.
  • Supervised features still have an edge over existing self-supervised approaches on NeWT.
  • iNat2021 and NeWT provide resources for benchmarking new techniques against challenges absent from standard datasets.

A.1. Dataset Construction

iNat2021 was constructed from an iNaturalist export through species filtering, selection, and observer-balanced test sampling. The resulting splits retain broad participation while limiting per-species training volume.

  • The source export contained over 19M research-grade observations across 200K species, with observations heavily concentrated among common species.50% of observations submitted in the previous year belonged to 1% of observed species.
  • The construction restricted candidates to Animalia, Plantae, and Fungi species meeting minimum observation and observer thresholds.
  • From 14k filtered species, the dataset retained the 10k species with the most observations before 9/25/2019.
  • Test images were sampled iteratively across observers until 50 observations per species were collected.
  • The validation split uses 10 random observations per species, while training contains at most 300 observations per species.
  • Train, mini-train, validation, and test splits include 158873, 79172, 32594, and 94061 observers, respectively.These observers averaged 12.8, 6.3, 3.1, and 5.2 unique species across the corresponding splits.

A.2. Data Quality

iNat2021 inherits label and image-content ambiguities from community observations. The authors retain this diversity without additional filtering or validation, while using NeWT for cleaner evaluation tasks.

  • Research-grade community labels contain error, with a small analysis matching expert identification about 85% of the time.The authors note uncertainty about whether archive-wide accuracy is closer to 95% or 85%.
  • Research-grade observations may depict indirect evidence such as footprints, scat, nests, burrows, or markings rather than the organism itself.
  • Images can represent different life stages, including alive versus dead and larval versus adult organisms.
  • The dataset retains these image modalities without filtering train or test splits or controlling their distribution within species.
  • Exactly one species label is assigned even when multiple species are prominently present, such as a bee pollinating a flower.The authors report multiple top-k results and introduce NeWT binary tasks to address this nuisance.

A.3. Object Size Statistics

Figure 5 compares cumulative bird bounding-box area distributions across CUB, NABirds, iNat2017, and ImageNet. The cited passage reports similar distributions for CUB, NABirds, and ImageNet.

  • Figure 5 plots the cumulative fraction of ground-truth bird bounding boxes as a fraction of image size for CUB, NABirds, iNat2017, and ImageNet.
  • CUB, NABirds, and ImageNet have very similar bird bounding-box distributions, with 50% of boxes exceeding 40% of image area.

A.4. iNat2021 Geographical Distribution

iNat2021 is geographically more diverse than its 2018 variant and includes metadata about image locations. Its bird images also contain much smaller subjects than several comparison datasets, a difference that may affect pretraining transfer.

  • Geographical distribution: iNat2021 training images are more geographically varied than those in the 2018 dataset variant.The dataset contains images from a larger number of countries, partially addressing the North American bias of datasets such as iNat2018.
  • Bird-scale distribution: Approximately 50% of iNat2017 bird bounding boxes have areas below approximately 7% of image size, unlike the larger birds in CUB, NABirds, and ImageNet.The iNat2017 distribution is intended to represent birds in iNat2021.
  • Bird-scale distribution: The bird-size discrepancy may explain why ImageNet is better pretraining than iNat2021 for CUB and NABirds with ImageNet-designed self-supervised techniques.
  • Geographical distribution: This broader geographic distribution enables more realistic exploration of diverse geographical distributions in problems such as federated learning.

B. Additional Results

Additional results compare representation methods across fine-grained datasets and NeWT tasks, while dataset analysis indicates that iNat2021 is more geographically varied than its 2018 variant.

  • Geographical distribution: iNat2021 is more geographically varied than the 2018 dataset variant, with no major observed difference between its training and test-image distributions.The test images were sampled between September 2019 and September 2020.

C. Additional Implementation Details

The experiments use author-provided implementations and fixed ResNet50 features evaluated with linear SVMs, with solver choice determined by dataset size. NeWT examples are center-cropped and resized for visualization.

  • Model setup: The study uses code repositories and available checkpoints from the respective authors, with default parameters for self-supervised iNat2021 models unless stated otherwise.
  • Model setup: Only iNat2021 mini is used for self-supervised training in these experiments to reduce time and cost.
  • Classifier training: The fine-grained experiments use linear SVMs trained with SGD, whereas NeWT uses linear SVMs trained with liblinear.
  • Classifier training: Liblinear is used for NeWT because its tasks contain at most 200 training features, while larger fine-grained datasets require SGD.
  • Visualization: NeWT example images are center-cropped and reduced in size for visualization.
  • Evaluation: Tables 5 and 6 evaluate ResNet50 pretrained on different source datasets as fixed feature extractors on fine-grained datasets and NeWT.
Loading 2103.16483v2…