PaperlayerView exact revision on arXiv

Source-linked AI summary

LVIS: A Dataset for Large Vocabulary Instance Segmentation

Agrim Gupta, Piotr Dollár, Ross Girshick

arXiv:1908.03195v2cs.CV

TL;DR

LVIS addresses the lack of rigorous benchmarks for instance segmentation with large vocabularies and scarce per-category examples. It introduces a federated dataset with high-quality masks and COCO-style evaluation, supporting more than 1000 categories across 164k images. The authors position LVIS as a benchmark for studying low-shot learning and large-vocabulary recognition.

  • Problem

    Rigorous object-detector evaluation rarely covers many categories with scarce per-category data, while fair evaluation and feasible annotation become challenging beyond 1000 categories.

  • Method

    LVIS combines COCO-style instance segmentation and average precision with a federated dataset whose category-specific subsets provide exhaustive annotations without labeling every category in every image.

  • Results

    LVIS is designed to span 164k images and approximately 2 million high-quality object instances, with masks reported as surpassing related datasets in annotation quality.

  • Takeaways & Limitations

    LVIS establishes a benchmark intended to enable rigorous study of large-vocabulary instance segmentation and low-shot learning.

  • Takeaways & Limitations

    Categories present in validation and test may be a strict subset of those in training, because the standard COCO 2017 splits do not guarantee category coverage across splits.

Abstract

from arXiv · show

Progress on object detection is enabled by datasets that focus the research community's attention on open challenges. This process led us from simple images to complex scenes and from bounding boxes to segmentation masks. In this work, we introduce LVIS (pronounced `el-vis'): a new dataset for Large Vocabulary Instance Segmentation. We plan to collect ~2 million high-quality instance segmentation masks for over 1000 entry-level object categories in 164k images. Due to the Zipfian distribution of categories in natural images, LVIS naturally has a long tail of categories with few training samples. Given that state-of-the-art deep learning methods for object detection perform poorly in the low-sample regime, we believe that our dataset poses an important and exciting new scientific challenge. LVIS is available at http://www.lvisdataset.org.

1. Introduction

LVIS targets a gap in rigorous evaluation for large-vocabulary instance segmentation, where categories are numerous and per-category data can be scarce. It combines a federated dataset design with high-quality masks to support this setting.

  • Motivation: Existing rigorous object-detector evaluation largely uses few categories or many examples per category, leaving large-vocabulary, low-shot settings underexplored.The long tail of rare categories makes learning from few examples a significant open problem.
  • Dataset: LVIS targets more than 1000 entry-level categories across 164k images with approximately 2 million high-quality instance masks.The dataset is intended as a benchmark for Large Vocabulary Instance Segmentation.
  • Dataset: Its annotation pipeline discovers naturally occurring long-tail categories through iterative object spotting without using machine-learning algorithms for labeling.Images are collected without prior knowledge of the categories that will be labeled.
  • Dataset: LVIS masks have higher overlap and boundary consistency than masks from COCO and ADE20K when compared with expert annotations.This quality target is intended to distinguish predicted mask quality beyond coarse annotation limits.
  • Evaluation and design: The benchmark uses COCO-style instance segmentation and average precision, preserving task and metric continuity while addressing multiple valid labels and annotation workload.These challenges motivate the federated dataset design, whose constituent datasets provide exhaustive annotations for individual categories.

2. Dataset Design

LVIS uses an evaluation-first federated dataset design to support fair instance-segmentation evaluation across many categories without exhaustively labeling every image. Per-category positive and negative image sets reduce annotation workload while preserving COCO-style AP evaluation.

  • Task and metric: The benchmark task is COCO-style instance segmentation evaluated with the COCO mask average precision metric.The design preserves task and metric continuity with COCO.
  • Evaluation challenges: Missing labels can unfairly penalize correct predictions when objects have multiple, hierarchical, or synonymous categories.Examples include toy deer labeled only as toy, or car labeled only as vehicle.
  • Federated evaluation: For each category c, the positive set Pc exhaustively annotates c, while the negative set Nc contains no instance of c.These disjoint sets provide the guarantees needed for per-category evaluation.
  • Federated evaluation: The federated dataset D = ∪c(Pc ∪Nc) evaluates category c only where its annotations are exhaustive or its absence is verified.Predictions outside Pc ∪Nc are not evaluated for that category.
  • Evaluation protocol: Test-set positive and negative membership is undisclosed, so algorithms must predict all categories without side information.This prevents systems from knowing which categories will be evaluated in each test image.
  • Workload: Federated design reduces |Pc ∪Nc| far below |D| for each category, lowering workload and allowing frequent categories to be undersampled.No single category is expected to exceed approximately 3% of the estimated 2 million instances.

3. Dataset Construction

LVIS constructs its vocabulary and annotations through an iterative crowdsourced pipeline that discovers categories, exhaustively marks instances, produces detailed masks, verifies quality, and defines negative sets.

  • Object spotting: Object spotting iteratively discovers categories and forms an initial positive set Pc for each vocabulary category.Images are revisited to find previously unmarked categories, and categories spotted in both runs are retained.
  • Instance marking: Exhaustive instance marking verifies each spotted category and marks all of its instances in each positive-set image.Five annotators review each image-category pair, with frequent categories subsampled to control workload.
  • Segmentation and verification: Instance segmentation converts marked objects into detailed masks, while segment verification iterates rejection and re-annotation to enforce quality.Up to five annotators judge each mask, and the process accepts a mask only when four agree it is high-quality.
  • Segmentation and verification: >99% of marked objects receive a high-quality segmentation mask after the stage 3–4 iterations.The pipeline alternates segmentation and verification four times.
  • Exhaustiveness: Full-recall verification flags positive-set images that are not exhaustively annotated for their category.Up to five annotators assess exhaustiveness, requiring four to agree that annotation is complete.
  • Negative sets: Negative sets Nc are built by sampling images outside Pc and rejecting any image where an annotator detects category c.Sampling continues until Nc reaches 1% of the dataset.
  • Vocabulary construction: The vocabulary begins with 8.8k concrete WordNet synsets and is narrowed through object spotting on COCO images.This process is intended to produce a high-recall vocabulary of visual, entry-level categories.
  • Dataset diversity: LVIS exhibits greater spatial complexity than COCO and Open Images v4, while ADE20K has the greatest spatial diversity.The comparison uses normalized object-center distributions.

4. Dataset Analysis

LVIS analysis finds a broad, long-tailed category distribution, highly repeatable annotations, and evaluation behavior that remains compatible with COCO while exposing low-shot difficulty.

  • Dataset statistics: 977 categories appear in 5,000 images, with 11.2 instances from 3.4 categories per image on average.The maximum observed count is 294 instances per image, and category coverage is more dispersed than COCO’s.
  • Dataset statistics: The number of instances per category reveals a long tail with few examples, while category counts per image have a heavier tail than COCO and Open Images.ADE20K’s category-count distribution is more uniform.
  • Annotation consistency: 0.87 average F1 score measures consistency of category labeling across two annotation runs.Roughly 75% of image-category pairs have a perfect F1 score.
  • Annotation consistency: 0.85 average mask IoU measures consistency between two independent LVIS annotation runs.The comparison uses matched instances from 5,000 doubly annotated images.
  • Annotation quality: LVIS has the highest quality across the reported mask IoU and boundary-quality measures in comparison with expert annotations.The figure reports bootstrapped 95% confidence intervals for means and medians.
  • Annotation consistency: Roughly 50% of matched instances have IoU greater than 90%, supporting substantial repeatability of the pipeline.The matching is established using a low IoU threshold of 0.5.
  • Evaluation validation: COCO-trained Mask R-CNN models produce similar box and mask AP on LVIS and original COCO annotations for COCO categories.The result supports the validity of LVIS annotations and its federated evaluation protocol.
  • Low-shot detection: 36.4% mask AP on the full COCO training set falls to 9.8% with 1k training images, demonstrating low-shot detection difficulty.In the 1k subset, 89% of categories still have more than 20 training instances.

5. Conclusion

LVIS introduces a large-scale benchmark for instance segmentation over more than 1,000 categories under low-shot conditions. The dataset combines broad image coverage with approximately 2 million high-quality masks.

  • Contribution: LVIS targets rigorous instance-segmentation research over more than 1,000 object categories under low-shot learning conditions.The benchmark is designed for algorithms that recognize a large vocabulary while coping with few examples.
  • Scale and quality: The planned dataset spans 164k images and approximately 2 million object instances with high-quality segmentation masks.The paper states that LVIS masks surpass the annotation quality of related datasets.
  • Benchmark role: LVIS is intended to become a benchmark challenge for object detection, segmentation, and low-shot learning algorithms.The paper presents this as a planned use of the dataset.

A. LVIS Release v0.5

LVIS release v0.5 expands the dataset to 82k annotated images and 1230 quality-controlled categories. Its train and validation sets remain statistically aligned, while most categories still have limited training data.

  • Release scope: 82k annotated images comprise release v0.5, including 57k training and 20k test images added beyond the 5k validation set.The release marks the halfway point in data collection.
  • Release scope: 1230 categories remain after quality control, following removal of visually inconsistent categories and categories absent from training.
  • Dataset consistency: The train and test sets are statistically identical because they were sampled from the same image distribution and mixed within annotation batches.
  • Dataset statistics: Approximately 75% of categories appear in 100 training images or fewer, preserving LVIS’s challenging low-shot distribution.
  • Dataset consistency: Train and validation object-center distributions are visually similar, indicating quality control did not introduce meaningful bias in this statistic.
  • Dataset consistency: The authors conclude that the data-collection process scales well beyond the initial 5k-image analysis set.

B. LVIS v0.5 Baselines

The LVIS v0.5 baseline section calibrates results for the LVIS Challenge using Mask R-CNN, rare-category resampling, and larger-model enhancements.

  • Baseline suite: LVIS v0.5 baselines evaluate out-of-the-box Mask R-CNN, repeat-factor resampling for rare categories, and larger models.
  • Baseline suite: The baseline suite is intended to help researchers calibrate results for the upcoming LVIS Challenge.

B.1. Mask R-CNN Out-of-the-Box

Out-of-the-box Mask R-CNN performs poorly on rare LVIS categories, but inference adjustments and stronger training configurations improve performance. LVIS also exhibits unusually strong mask results relative to box results.

  • Configuration: Mask R-CNN with a standard ResNet-50-FPN configuration uses an unmodified COCO training formula.
  • Model enhancements: Scale jitter and larger backbones improve all AP metrics, although APr does not improve with the largest backbone.
  • Out-of-the-box results: 0.8% APr for rare categories demonstrates that out-of-the-box Mask R-CNN performs near zero in the low-shot regime.
  • Inference adjustments: Increasing detections per image to 300 and removing the minimum score threshold raises APr to 3.2%.
  • Mask and box evaluation: LVIS mask AP is typically slightly higher than box AP, opposite the 3–4% absolute mask-versus-box gap typically observed on COCO.

B.2. Mask R-CNN with Data Resampling

LVIS’s repeat-factor sampling oversamples images containing rare categories and substantially improves rare-category performance, with a smaller overall AP gain. Class-aware sampling helps rare categories but reduces common and frequent-category performance.

  • Repeat-factor sampling: Repeat-factor sampling oversamples images containing tail categories to increase their training frequency.
  • Repeat-factor sampling: For each image, the sampler uses the maximum repeat factor among its labeled categories and repeats images accordingly during each epoch.
  • Repeat-factor sampling: The threshold t controls when oversampling begins, while the method applies an inverse-frequency heuristic above that threshold.
  • Repeat-factor results: 13.4% APr at t = 0.001 improves rare-category AP from the 3.2% baseline, while overall AP increases 2.2%.The setting oversamples categories appearing in less than 0.1% of images.
  • Class-aware sampling: Class-aware sampling raises APr from 3.2% to 8.5%, but decreases APc and APf enough to worsen the overall result.

B.3. Mask R-CNN Standard Enhancements

Standard enhancements combine repeat factor sampling with scale jitter and larger models, yielding a final validation AP of 27.1%.

  • Scale jitter samples the training image scale from {640, 672, 704, 736, 768, 800}.
  • Larger models are applied alongside repeat factor sampling with t = 0.001.
  • 27.1% final validation AP is achieved after applying both enhancements.

C. LVIS val to test Results Transfer

LVIS evaluation-set size affects both category composition and AP transfer because larger sets contain more rare categories. The val-to-test shift is predicted empirically and can lower overall AP by about 2%.

  • Larger evaluation sets contain a higher proportion of categories that are rare in the training set.
  • 15.1% rare categories occur in the 5k-image val set versus 28.2% in the 20k-image test set.
  • The val and test rare-category proportions are predicted as ∼15% and ∼29%, respectively, before observing actual values.
  • ∼2% overall AP decrease may result when moving from val to test under the observed category-distribution shift.
  • Rare categories play a more important role on the test set than on the val set.

C.2. AP as a Function of Evaluation Set Size

AP estimates can depend systematically on evaluation-set size, with smaller sets producing higher AP, especially when categories are rare. LVIS therefore shows a val-to-test AP bias even though model rankings can transfer.

  • AP as a Function of Evaluation Set Size: Smaller evaluation subsets consistently produce higher AP than larger subsets for fixed classifiers across category frequencies.
  • AP as a Function of Evaluation Set Size: AP is lower for rarer categories at every evaluation size in both simulated classifier settings.
  • AP as a Function of Evaluation Set Size: The same evaluation-size trend appears for trained detectors evaluated on COCO val2017 subsets.
  • AP as a Function of Evaluation Set Size: Well-sampled COCO categories have largely converged by the 5k-image val2017 set, so overall AP changes little between val2017 and test2017.
  • AP as a Function of Evaluation Set Size: LVIS test subsets from 5k to 20k images reproduce the predicted AP changes, with the 5k subset aligning with the 5k-image val set.
  • AP as a Function of Evaluation Set Size: Model rankings transfer from val to test for the three models compared in Table 4b.
  • AP as a Function of Evaluation Set Size: The reported bias is attributed to at least two factors associated with greater class imbalance, producing higher val AP than test AP.
Loading 1908.03195v2…