PaperlayerView exact revision on arXiv

Source-linked AI summary

LVIS: A Dataset for Large Vocabulary Instance Segmentation

Agrim Gupta, Piotr Dollár, Ross Girshick

arXiv:1908.03195v2cs.CV

TL;DR

Object-detection evaluation has largely focused on few categories or abundant per-category examples, leaving the many-category, scarce-example setting open. LVIS proposes a federated instance-segmentation benchmark, whose masks are reported to surpass the annotation quality of related datasets.

  • Problem

    Object-detection evaluation mostly covers few categories or many examples per category, leaving the natural setting of many categories and scarce per-category data open for study.

    Source: §1 · p. 1

  • Method

    LVIS forms a federated dataset by uniting per-category datasets with exhaustive annotations within each category, reducing the need to annotate every image for every category.

    Source: §1 · p. 2, §2.2 · p. 4

  • Results

    LVIS's planned masks for more than 1000 categories are reported to surpass the annotation quality of related datasets.

    Source: §5 · p. 8

  • Takeaways & Limitations

    LVIS is intended as a benchmark challenge for instance-segmentation algorithms that recognize more than 1000 categories while coping with low-shot learning.

    Source: §5 · p. 8

    The standard COCO 2017 validation and test splits may contain a strict subset of training categories, and LVIS cannot guarantee category overlap across these splits.

    Source: §2.3 · p. 4

Abstract

from arXiv · show

Progress on object detection is enabled by datasets that focus the research community's attention on open challenges. This process led us from simple images to complex scenes and from bounding boxes to segmentation masks. In this work, we introduce LVIS (pronounced `el-vis'): a new dataset for Large Vocabulary Instance Segmentation. We plan to collect ~2 million high-quality instance segmentation masks for over 1000 entry-level object categories in 164k images. Due to the Zipfian distribution of categories in natural images, LVIS naturally has a long tail of categories with few training samples. Given that state-of-the-art deep learning methods for object detection perform poorly in the low-sample regime, we believe that our dataset poses an important and exciting new scientific challenge. LVIS is available at http://www.lvisdataset.org.

1. Introduction

LVIS is designed to make large-vocabulary instance segmentation research possible in the natural long-tail setting, where many categories have scarce training data. It combines a large-scale dataset with a federated design for fair evaluation.

1. Introduction

  • The long tail persists as more images reveal rare categories, while current detector evaluations largely use few categories or abundant examples, motivating a high-quality benchmark for scarce-data research.The paper identifies learning from few examples as an open problem in machine learning and computer vision.Source: §1 · p. 1
  • LVIS plans instance segmentation masks for more than 1000 entry-level categories in 164k images, with approximately 2 million high-quality masks projected after labeling 85k images.The introduction reports LVIS v0.5 as 77k additionally annotated images, about 50% of the planned dataset; detailed analysis uses a 5000-image validation subset annotated twice.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • An iterative object-spotting pipeline discovers categories without machine-learning-assisted labeling, and the resulting masks have higher overlap and boundary consistency than COCO [23] and ADE20K.The pipeline starts with images collected without prior knowledge of which categories will be labeled; the paper argues fine masks enable finer discrimination of predicted mask quality.Source: §1 · pp. 1–2
  • LVIS uses COCO-style instance segmentation and average precision across categories and mask IoU thresholds; its federated design requires exhaustive labels within each category dataset, not across every image.At test time, algorithms do not know image membership in constituent datasets and must predict as if all categories will be evaluated; the evaluation oracle scores each category on its constituent dataset.Source: §1 · p. 2

1.1. Related Datasets

  • Unlike COCO [23]’s 80 exhaustive categories and ADE20K ’s top-100 benchmark, LVIS targets large-vocabulary segmentation; unlike iNaturalist [34] and Open Images [18], it emphasizes entry-level masks and low-shot categories.iNaturalist lacks segmentation masks and focuses on fine-grained categories; Open Images evaluates frequent categories with many training samples and uses machine-learning-based image selection.Source: §1.1 · pp. 2–3

2. Dataset Design

LVIS uses an evaluation-first design for large-vocabulary instance segmentation, where overlapping, hierarchical, and synonymous categories make fair evaluation difficult. Its federated dataset evaluates categories on suitable image subsets, reducing annotation workload while defining explicit rules for incomplete annotations and category relationships.

2. Dataset Design

  • Before collecting data, LVIS defined instance segmentation and mask average precision (AP) evaluation because large vocabularies introduce challenges that affect dataset structure.The task asks for masks, category labels, and confidence scores for instances in unseen images; mask AP follows COCO.Source: §2 · p. 3, §2.1 · p. 3

2.1. Task and Evaluation Overview

  • Large vocabularies create overlapping, parent-child, and synonymous category relationships that can unfairly penalize correct detections when ground-truth labels are missing.Exhaustively labeling every category in every image would be costly and could force ambiguous judgments with inter-annotator disagreement.Source: §2.1 · pp. 3–4

2.2. Federated Datasets

  • For each category, LVIS uses an exhaustively annotated positive image set and a negative set, evaluating detections only on their union and leaving other images unevaluated for that category.The test split does not disclose set membership, so algorithms must predict all categories in every test image.Source: §2.2 · p. 4
  • Federating category-specific datasets reduces annotation workload and permits undersampling frequent categories; no category is expected to exceed about 3% of the estimated 2 million instances.The per-category image subsets can be much smaller than the full dataset, avoiding wasted annotation effort on common categories such as person.Source: §2.2 · p. 4

2.3. Evaluation Details

  • LVIS reports only overall category-averaged AP; it treats hierarchy uniformly, expects predictions to include the most specific correct label and its general categories, and uses WordNet synsets to group synonyms.For images marked non-exhaustive (9% of cases), false positives are not counted, but recall is measured for labeled instances; overall AP avoids category leakage and unstable rare-category scores. The val and test category sets may be strict subsets of the training categories.Source: §2.3 · p. 4
  • The six-stage annotation pipeline spots objects, exhaustively marks and segments instances, verifies segmentation quality and completeness, then assigns negative labels.Segmentation and verification repeat until about 99% of segmentations pass a quality check.Source: Figure 4 · p. 5

3. Dataset Construction

LVIS uses a staged annotation pipeline to turn spotted objects into verified masks and exhaustive positive and negative sets, while iteratively building a 1,723-category vocabulary.

3.1. Annotation Pipeline

  • Object spotting iteratively elicits category names, retains categories found in two independent runs, and creates an initial positive image set for each vocabulary category.Annotators mark objects and name them using synonym-aware autocomplete; an image is retired after three skips, and two runs improve naming consistency.Source: §3.1 · p. 5
  • Five annotators verify each spotted image-category pair and mark all instances; gold-set checks support quality control, while frequent-category positive sets are capped at 1% of images.The five annotators help ensure instance-level recall, and categories that fail verification are not sent to exhaustive instance marking.Source: §3.1 · p. 5
  • More than 99% of marked objects receive high-quality masks after segmentation and verification, with rejected instances re-annotated through four iterations.Annotators verify each object's category and draw a detailed mask; up to five reviewers assess mask quality, and at least four must accept it.Source: §3.1 · pp. 5–6
  • Full-recall checks finalize positive sets with an exhaustiveness flag, while category-specific negative sets contain sampled images confirmed to lack that category.At least four annotators must agree an image is exhaustively annotated; a negative image is rejected if any annotator reports the category present, and sampling targets 1% of dataset images.Source: §3.1 · p. 6

3.2. Vocabulary Construction

  • The vocabulary process winnows 8.8k WordNet synsets through spotting on 10k COCO images and minor manual editing to 1,723 synsets.The resulting vocabulary sets the upper bound on categories that can appear in LVIS; the initial synsets were selected for concrete common nouns likely to be visual.Source: §3.2 · p. 6
  • LVIS has greater spatial complexity than COCO and Open Images v4, while ADE20K shows the greatest spatial diversity.The comparison concerns distributions of object centers in normalized image coordinates.Source: Figure 5 · p. 6

4. Dataset Analysis

The analysis of 5,000 doubly annotated COCO val2017 images characterizes LVIS’s category, spatial, and scale distributions, annotation consistency, and evaluation protocol.

4.1. Dataset Statistics

  • Across 5,000 images, LVIS has 977 categories (median 9 instances per category), more spread in categories per image than COCO, more diverse object-center locations, and about 10× fewer image-filling objects.Each image averages 11.2 instances from 3.4 categories; LVIS objects tend to be smaller than COCO objects.Source: §4.1 · p. 6, Figure 6 · p. 7

4.2. Annotation Consistency

  • Across two annotations, mean mask IoU is 0.85 and mean F1 is 0.87; roughly 50% of matched masks exceed 90% IoU and 75% of image-category pairs have perfect F1.Matches were formed at IoU ≥ 0.5; the results indicate substantial pipeline repeatability.Source: §4.2 · p. 7, Figure 7 · p. 7
  • Compared with COCO and ADE20K, LVIS masks score higher on mean and median mask IoU and boundary quality, despite having more complex boundaries.The expert comparison uses bootstrapped 95% confidence intervals; boundary complexity is perimeter divided by square root area.Source: §4.2 · p. 7, §4.1 · p. 7, Table 1 · p. 7

4.3. Evaluation Protocol

  • Box and mask AP from COCO-trained Mask R-CNN models are close between LVIS and COCO annotations, validating the annotations and federated evaluation protocol.The comparison covers three models spanning a wide AP range.Source: §4.3 · p. 8, Table 2 · p. 8
  • With 80 positive examples, simulated AP is similar to baseline; increasing negatives lowers AP by about 2% absolute, while five positives raise variance but keep the interquartile AP gap below 2%.These simulations support using smaller per-category evaluation sets.Source: §4.3 · p. 8, Figure 8 · p. 8
  • Mask AP falls from 36.4% on the full dataset to 9.8% on a 1k-image subset, despite 89% of that subset’s categories having over 20 training instances.Categories are classified by training-set image count as rare (1–10), common (11–100), or frequent (>100); the analysis projects over 1,000 categories in the final dataset, with low-shot categories remaining prevalent as it grows.Source: §4.3 · p. 8, Figure 8 · p. 8, Figure 9 · p. 8

5. Conclusion

LVIS is designed to support rigorous study of instance segmentation for more than 1,000 categories under low-shot learning. It combines large-scale image and instance coverage with high-quality masks and is intended to become a benchmark challenge.

5. Conclusion

  • LVIS targets instance segmentation across more than 1,000 categories using methods that can cope with low-shot learning.The dataset is designed to enable rigorous study of these algorithms.Source: §5 · p. 8
  • LVIS is planned to span 164k images and label approximately 2 million object instances with high-quality masks that surpass related datasets’ annotation quality.Source: §5 · p. 8
  • The authors hope to establish LVIS as a benchmark challenge that spurs new object detection, segmentation, and low-shot learning algorithms.Source: §5 · p. 8

A. LVIS Release v0.5

LVIS release v0.5 expands the annotated dataset to 82k images and 1230 categories after quality control. Its training set remains long-tailed, while train and validation object-center distributions are nearly identical.

A. LVIS Release v0.5

  • Quality control removed about 18% of categories and 10% of labeled instances; release v0.5 contains 1230 categories.The validation category count fell from 977 to 830, and the release includes 51k validation and 694k training segmented instances.Source: A. LVIS Release v0.5 · p. 9
  • The release adds 77k annotated images to the prior 5k validation set, bringing the total to 82k annotated images.The added images comprise 57k training and 20k test images; v0.5 marks the halfway point in data collection.Source: A. LVIS Release v0.5 · p. 9
  • Training and test data are statistically identical because both were sampled from the same image distribution and mixed in each annotation batch.The train and test sets were collected using the same process.Source: A. LVIS Release v0.5 · p. 9
  • Train and validation object-center distributions are nearly identical, and quality control did not meaningfully bias this spatial statistic.Figure 11 compares LVIS val, val v0.5 after quality control, and train v0.5.Source: Figure 11 · p. 9, A. LVIS Release v0.5 · p. 9
  • Roughly 75% of categories appear in 100 training images or fewer, so v0.5 retains a substantial low-shot challenge.The proportion of rare categories is decreasing as collection continues, but remains sizeable.Source: A. LVIS Release v0.5 · p. 9
  • The authors conclude that the data-collection process scales well beyond the initial 5k-image set.This conclusion follows the expanded release analysis and per-category quality control.Source: A. LVIS Release v0.5 · p. 9

B. LVIS v0.5 Baselines

To calibrate results for the upcoming LVIS Challenge at ICCV 2019, the section introduces simple baselines spanning inference-time adjustments, rare-category resampling, and larger models.

B.1

  • Mask R-CNN is tested out of the box, highlighting the importance of adjusting two inference-time hyper-parameters.Source: B. LVIS v0.5 Baselines · p. 9

B.2

B.3

B.1. Mask R-CNN Out-of-the-Box

Out-of-the-box Mask R-CNN performs poorly on LVIS, especially for rare categories; inference-time changes modestly improve rare-category AP, while augmentation and larger backbones improve most AP metrics.

B.1. Mask R-CNN Out-of-the-Box

  • The baseline uses ImageNet-pretrained ResNet-50 with FPN and an unmodified COCO training formula, trained in Detectron2.Training uses a 90k-update schedule, horizontal flipping only, 800-pixel shorter-edge resizing, and no test-time augmentation.Source: §B.1 · p. 9
  • Scale jitter and larger backbones improve all AP metrics, except that APr does not improve with the largest backbone.The largest-backbone exception applies specifically to rare-category mask AP.Source: §B.1 · p. 10
  • APr rises from 0.8% to an average of 3.2% after increasing detections per image and removing the score threshold; the changes also substantially improve APc.LVIS permits up to 300 detections per image, and the minimum confidence threshold is reduced from 0.05 to 0.0.Source: §B.1 · p. 10, Table 3 · p. 10
  • LVIS mask AP is typically slightly higher than box AP, unlike COCO, where AP is typically 3 to 4% lower; degrading masks drops AP with almost no change in box AP.The authors hypothesize that LVIS’s high-quality segmentation masks explain this pattern.Source: §B.1 · p. 10

B.2. Mask R-CNN with Data Resampling

The section evaluates repeat factor sampling and class-aware sampling (CAS) for Mask R-CNN on imbalanced categories. Repeat factor sampling oversamples images containing tail categories, improving rare-category AP and overall AP at a selected threshold.

B.2. Mask R-CNN with Data Resampling

  • Repeat factor sampling oversamples images containing tail categories to increase how often those categories are observed during training.Each image’s repeat factor is the maximum repeat factor among its labeled categories, and the sampler repeats images accordingly in each epoch.Source: §B.2 · p. 10
  • The method’s sole hyperparameter t sets when oversampling begins: categories with fc ≤ t are not oversampled, while those with fc > t follow a square-root inverse-frequency heuristic.When a category’s frequency decreases by a factor γ < 1, its repeat factor is multiplied by 1/γ.Source: §B.2 · p. 10
  • At t = 0.001, repeat factor sampling raises APr from 3.2% to 13.4% and overall AP by 2.2%, with a 0.6% decrease in APf.This threshold oversamples categories appearing in less than 0.1% of images (829 of 1230 categories).Source: §B.2 · p. 10
  • CAS raises APr from 3.2% to 8.5%, but decreases both APc and APf, worsening overall results.CAS samples a category and then an image containing it, iterating over random permutations of categories and their images.Source: §B.2 · p. 10

B.3. Mask R-CNN Standard Enhancements

With repeat factor sampling at t = 0.001, scale jitter and larger models improve Mask R-CNN, yielding a final validation AP of 27.1%.

B.3. Mask R-CNN Standard Enhancements

  • 27.1% validation AP is achieved by adding scale jitter and larger models to repeat factor sampling with t = 0.001.Training scale is sampled from {640, 672, 704, 736, 768, 800}; both enhancements improve results.Source: §B.3 · p. 10

C. LVIS val to test Results Transfer · C.1. Category Frequency Distributions

Larger evaluation sets contain a higher proportion of rare categories: LVIS test has 28.2% rare categories versus 15.1% on val, a shift that could lower overall AP by about 2%.

C. LVIS val to test Results Transfer

  • The transfer analysis reports that smaller evaluation sets can bias AP upward, especially for rare categories, so LVIS test AP is lower than val AP; the effect is minimal on COCO.Source: C. LVIS val to test Results Transfer · p. 11

C.1. Category Frequency Distributions

  • Evaluation sets with more categories that appear infrequently in training tend to be harder because learning from few examples is challenging.Source: §C.1 · p. 11
  • The analysis estimates mean rare, common, and frequent category shares across random image subsets of different sizes.Category frequency groups are defined by how many training images contain each category.Source: §C.1 · p. 11
  • A shift toward rare categories could decrease overall AP by ∼2% from val to test, making rare categories more important in the test set.The estimate uses the APr/c/f achieved by R-101-FPN Mask R-CNN as an example.Source: §C.1 · p. 11
  • The predicted rare-category shares—∼15% for val and ∼29% for test—closely match observed values of 15.1% and 28.2%.Source: §C.1 · p. 11
  • Larger evaluation sets generally contain a higher proportion of categories that are rare in the training set.Source: §C.1 · p. 11

C.2. AP as a Function of Evaluation Set Size · C.3. Comparing Models · C.4. Summary

In LVIS, AP estimates from smaller evaluation sets are biased upward, although the tested models retain their validation-set ranking on test. The analysis also finds statistical evidence for a 0.5-point test AP difference and motivates a larger validation set for a future release.

C.2. AP as a Function of Evaluation Set Size

  • On COCO, APbb@75 for toaster, cats, and dining table showed similar subset-size trends, but well-sampled cats and dining table had converged by 5k images.Consequently, overall COCO AP varies little between val2017 and test2017.Source: §C.2 · p. 11
  • In both Gaussian-score and real-classifier simulations, AP is consistently higher for smaller evaluation subsets across category frequencies.For a fixed classifier, rarer categories also have lower AP.Source: §C.2 · p. 11, Figure 13 · p. 12
  • On LVIS, fixed Mask R-CNN AP systematically decreases as evaluation subsets grow from 5k to 20k images, with 5k results aligning with val v0.5.The paper attributes the expected difference between val and test to LVIS's large number of rare categories.Source: Table 4 · p. 12, §C.2 · p. 12
  • The three models' ranking on val transferred exactly to test, despite the expected AP bias between evaluation sets of different sizes.The comparison covered three models with distinctly different validation AP.Source: Table 4 · p. 12, §C.2 · p. 12

C.3. Comparing Models

  • For ResNet-101 versus ResNeXt-101 (test AP 20.0 vs. 20.5), paired t-test and permutation-test p-values were 0.0490 and 0.0486, and the 95% bootstrap interval [0.002, 1.015] excluded 0.All three tests rejected a zero mean per-category AP difference at the 5% level.Source: §C.3 · p. 12

C.4. Summary

  • With greater class imbalance than class-balanced detection datasets, smaller LVIS validation sets yield higher AP than test; the authors propose extending v0.5's small validation set in release v1.The analysis identifies at least two factors behind the bias but this summary passage does not specify them.Source: §C.4 · p. 12

Related papers

Loading 1908.03195v2…