PaperlayerView exact revision on arXiv

Source-linked AI summary

PACO: Parts and Attributes of Common Objects

Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, Amir Mousavi, Yiwen Song, Abhimanyu Dubey, Dhruv Mahajan

arXiv:2301.01795v1cs.CV

TL;DR

Existing common-object datasets do not jointly provide large-scale part masks, object attributes, and part attributes. PACO combines LVIS and Ego4D data into a curated dataset with three benchmarks, reporting 641K part masks across 260K object instances and evaluation resources for segmentation, attribute prediction, and zero-shot detection.

  • Problem

    Large benchmark datasets for common objects lack joint annotations of part masks, object attributes, and part attributes, while existing part datasets are limited in instance scale.

  • Method

    PACO constructs image and video datasets with curated part and attribute vocabularies and benchmarks part segmentation, attribute detection, and zero-shot instance detection.

  • Results

    641K part masks span 260K object instances across 75 object classes, with 421K combined object and part masks carrying attribute annotations.

  • Takeaways & Limitations

    PACO provides a common-object benchmark for jointly evaluating object, part, and attribute understanding across image and video data.

  • Takeaways & Limitations

    Some valid object-part pairs present in training lack annotated instances in validation and test splits and are ignored during evaluation.

Abstract

from arXiv · show

Object models are gradually progressing from predicting just category labels to providing detailed descriptions of object instances. This motivates the need for large datasets which go beyond traditional object masks and provide richer annotations such as part masks and attributes. Hence, we introduce PACO: Parts and Attributes of Common Objects. It spans 75 object categories, 456 object-part categories and 55 attributes across image (LVIS) and video (Ego4D) datasets. We provide 641K part masks annotated across 260K object boxes, with roughly half of them exhaustively annotated with attributes as well. We design evaluation metrics and provide benchmark results for three tasks on the dataset: part mask segmentation, object and part attribute prediction and zero-shot instance detection. Dataset, models, and code are open-sourced at https://github.com/facebookresearch/paco.

1. Introduction

PACO addresses the lack of large common-object benchmarks that jointly annotate parts and attributes by introducing a dataset and three evaluation tasks for detailed object understanding.

  • Motivation: Existing datasets provide object-level attributes or limited part masks, but none jointly annotate part masks, object attributes, and part attributes for common objects.Part annotations are often domain-specific or limited in scale, while existing attribute datasets lack part-level attributes.
  • Dataset and contribution: PACO contains 641K part masks across 77K images and 260K object instances, covering 75 object classes, 456 object-specific part classes, and 55 attributes.The dataset combines image data from LVIS and video data from Ego4D, with quality-controlled vocabulary construction.
  • Benchmark tasks: The benchmark evaluates part segmentation, object and object-part attribute detection, and zero-shot instance detection using part- and attribute-based queries.The first two tasks measure standalone part and attribute understanding, while the third evaluates a downstream retrieval task.
  • Design choices: PACO evaluates parts and attributes as object-conditioned categories because the same semantic part can look different across object types.This design requires models to jointly predict objects, parts, and attributes rather than treating them as independent labels.
  • Evaluation resources: The work provides Average Precision and Average Recall metrics, benchmark baselines, and open-source dataset, models, and code.These resources are intended to calibrate future research models across PACO’s tasks.

2. Dataset construction

PACO combines LVIS images and Ego4D video frames, constructs curated part and attribute vocabularies, and uses staged annotations with quality control for masks, attributes, and instances.

  • Data sources: PACO uses LVIS for images and Ego4D for video, selecting 75 object categories common to both sources.The selected categories had at least 20 instances in Ego4D.
  • Vocabulary construction: Part names are mined from web images and manually curated to retain parts that are usually visible and clearly distinguishable for each object category.The pipeline is designed to provide an ontology of common-object parts rather than domain-specific parts.
  • Vocabulary construction: A user study produces an attribute vocabulary of 29 colors, 10 patterns and markings, 13 materials, and 3 reflectance levels.The study sought attributes capable of distinguishing object instances of the same type.
  • Annotation pipeline: The annotation pipeline covers object boxes and masks, part masks, object and part attributes, and Ego4D instance IDs.Object boxes and masks are annotated only for Ego4D because LVIS already provides them.
  • Annotation pipeline: Part masks are provided for annotated boxes, yielding 209K LVIS and 43K Ego4D object boxes with parts after rejecting low-resolution, blurred, or heavily occluded boxes.Annotators label visible parts from a class-specific list, and parts may overlap.
  • Attribute annotation: LVIS attributes are sampled from one medium or large box per image and object class, whereas Ego4D boxes receive object- and part-level attributes unless rejected for quality.The selected boxes are annotated for all 55 attributes in a single annotation process.
  • Instance identity and quality control: Ego4D instance IDs require multi-stage annotation because different boxes and videos can depict the same physical object.Quality control includes expert gold annotations, audits, and reannotation when class-level mask mIoU falls below 50%; ultimately, 90% of object classes reach mIoU ≥0.75 on gold masks.

3. Dataset statistics

PACO’s statistics show broad but long-tailed part coverage, smaller part masks than object masks, and substantially larger combined part-and-object attribute coverage than prior datasets.

  • Part statistics: Part categories follow a long-tail distribution, with some categories exceeding 6500 instances and others having fewer than 5.Examples of frequent categories include book-cover, chair-back, and box-side; fan-logo and kettle-cable are rare.
  • Part statistics: Part masks are more concentrated in low and medium sizes than object masks in PACO-LVIS.This size distribution helps explain why part segmentation is generally harder than object segmentation.
  • Dataset scale: 260K object instances with parts give PACO 10× more such instances than PartsImageNet’s 25K.PACO also provides a more extensive instance-level benchmark for parts than datasets with limited test annotations.
  • Attribute statistics: PACO contains 421K combined object and part masks with attributes, with attribute density 23.4 per image versus 3.6 for VAW.Unlike VAW’s larger 620-attribute vocabulary, PACO exhaustively annotates all 55 attributes for each selected object or part mask.

4. Tasks and evaluation benchmark

PACO defines three benchmarks for part segmentation, attribute prediction, and zero-shot instance detection, with evaluation procedures adapted to jointly scored object, part, and attribute labels.

  • The benchmark covers part segmentation, object and object-part attribute prediction, and zero-shot instance detection.
  • Part segmentation: Part segmentation detects and segments object-part instances while assigning a joint object-part label and confidence score.
  • Part segmentation: 4,562 object-part categories are evaluated separately because the same semantic part can differ across object categories.
  • Attribute prediction: Attribute prediction jointly scores object or object-part categories with attribute labels using federated AP calculations.
  • Zero-shot instance detection: Zero-shot queries describe instances with level-1, level-2, or level-3 object and part attributes, each paired with one positive image and up to 100 distractors.
  • Zero-shot instance detection: Queries avoid uninformative attributes and are evaluated with average recall at top-k returned boxes across IoU thresholds.

5. Benchmarking experiments

Experiments benchmark segmentation, attribute prediction, and zero-shot instance detection using Mask R-CNN and ViT-det variants. Larger models improve segmentation and attribute prediction, while attribute prediction remains harder and descriptive zero-shot detection remains challenging.

  • Mask R-CNN and ViT-det models are trained with 531 object and object-part classes and evaluated on PACO-LVIS splits.
  • Part segmentation: Object-part segmentation generally achieves lower AP than object segmentation because parts are typically smaller, while larger ViT-L backbones improve part segmentation.
  • Attribute prediction: Attribute prediction has lower AP than object and part detection, although larger models perform better.
  • Attribute prediction: A large gap between lower and upper attribute-score bounds shows that the models only partially bridge the attribute-prediction range.
  • Zero-shot instance detection: Zero-shot instance detection ranks boxes by combining object, part, object-attribute, and part-attribute scores with a geometric mean.
  • Zero-shot instance detection: Existing open-vocabulary detectors struggle with PACO's descriptive queries, highlighting an opportunity to handle object descriptions beyond category labels.
  • Few-shot comparison: A 20+ point gap separates the best zero-shot model from one-shot detection, and the gap widens as the number of shots increases.

6. Conclusion

PACO enables joint research on detecting common objects, their parts, and attributes across image and video data, with benchmarks covering segmentation, attribute prediction, and zero-shot detection.

  • 6. Conclusion: PACO provides part masks and attributes for 75 common object categories across image and video datasets.The dataset uses web-derived and manually defined part references to support annotation.
  • 6. Conclusion: The benchmark includes part segmentation, object and part attribute prediction, and zero-shot instance detection.These tasks address smaller part masks and instance-level attribute variation.
  • 6. Conclusion: Web images supply part vocabularies and pointers, while manually defined references cover objects for which web images are insufficient.The resulting references guide annotators in identifying object regions.

A.2. Attribute vocabulary selection

PACO selects attributes through user studies and greedy coverage analysis, retaining a compact vocabulary that distinguishes most object instance pairs while combining object- and part-level evidence.

  • A.2. Attribute vocabulary selection: The study evaluated color, shape, reflectance, materials, and patterns and markings for distinguishing paired instances.Annotators compared two instances of the same object using object and common-part regions.
  • A.2. Attribute vocabulary selection: Shape was removed because more than 50% of shape differences contained unnameable attributes that annotators found difficult to describe.The remaining attributes were considered sufficiently discriminative without shape.
  • A.2. Attribute vocabulary selection: Coverage is defined as the number of object pairs distinguishable by a selected set of object-level and part-level attributes.A greedy procedure adds the attribute distinguishing the largest number of previously unresolved pairs.
  • A.2. Attribute vocabulary selection: 98% of object instance pairs could be distinguished using the 55 attributes included in PACO.Coverage plateaued at 40 attributes, while the final taxonomy contains 55.
  • A.2. Attribute vocabulary selection: Object- and part-level attributes both contribute to instance discrimination, with 18% of pairs requiring only object attributes and 10% only part attributes.Color was the largest discriminative attribute type, distinguishing at least 75% of pairs through object and part color differences.
  • A.3.1 Instance annotation: Ego4D instance identities were constructed by splitting, embedding-based merging, and expert re-splitting of object occurrences.The process addressed repeated occurrences of the same instance across video frames.

B. Dataset annotation examples

PACO annotations pair object boxes with part masks and attributes, while dataset distributions and model configurations expose variation across categories and evaluation settings.

  • B. Dataset annotation examples: Object and part masks are used to crop segments for attribute annotations, with object boxes retaining visible attributes in full-image examples.Figures show object annotations on the left and part segmentation masks on the right.
  • B. Dataset annotation examples: Joint segmentation experiments evaluate Mask R-CNN and ViT-det models trained on PACO-LVIS and PACO-EGO4D.The reported tables cover evaluations on both datasets.
  • B. Dataset annotation examples: PACO-LVIS and PACO-EGO4D cover all 75 object classes and 71 classes with at least 10 instances, respectively.The least frequent categories contain 23 instances for drill in PACO-LVIS and 7 for scarf in PACO-EGO4D.
  • B. Dataset annotation examples: ViT-L model predictions are shown as qualitative examples of PACO outputs.These examples complement the quantitative segmentation and annotation displays.

D.1. Joint training on PACO-LVIS and PACO-EGO4D

Joint training on PACO-LVIS and PACO-EGO4D reveals lower overall performance on Ego4D, modest PACO-LVIS attribute gains, and backbone-dependent effects on object segmentation.

  • D.1. Joint training on PACO-LVIS and PACO-EGO4D: Performance on PACO-EGO4D is overall lower than on PACO-LVIS for jointly trained models.This indicates that the video dataset presents a greater challenge in the reported evaluations.
  • D.1. Joint training on PACO-LVIS and PACO-EGO4D: Joint image-and-video training improved PACO-LVIS attribute results by 0.2% for R50-FPN and R101-FPN and 0.8% for ViT-B FPN.These gains are reported relative to models trained only on PACO-LVIS.
  • D.1. Joint training on PACO-LVIS and PACO-EGO4D: Validation and test splits preserve consistent result rankings and similar trends across architectures.The study uses this consistency to assess whether validation results support model tuning.
  • D.3. Object segmentation only models: Joint training slightly deteriorates object segmentation for R-50 but considerably improves it for ViT-L.The ablation compares models trained only for object segmentation with models trained on multiple tasks.
  • D.1. Joint training on PACO-LVIS and PACO-EGO4D: Attribute-aware AP bounds use equal object and attribute scores for the lower bound and perfect attribute prediction for the upper bound.The bounds apply to both objects and object-parts.

E. Additional zero-shot instance detection results

Additional experiments evaluate zero-shot instance detection with FPN and cascade models, alongside broader PACO benchmark evaluations and transfer analyses.

  • Cascade models improve zero-shot instance detection performance for all but the largest model on PACO-LVIS.
  • The benchmark includes attribute prediction results for both object attributes and object-part attributes on PACO-EGO4D.
  • PACO evaluates object detection, object-part detection, and attribute prediction across validation and test splits.

F. Ablation studies for zero-shot instance detection

Ablations show that richer object, part, and attribute information improves zero-shot instance detection, while query scoring combines model outputs after associating detected parts with objects.

  • Each additional object, part, color, or attribute component improves instance detection performance over object-only predictions.
  • Query scoring first associates compatible detected parts with object boxes using class matching and mask containment.Matched parts require more than 50% of the part mask area to lie within the object mask, with the highest-scoring duplicate retained.
  • The scoring function combines object, object-attribute, part, and part-attribute scores for each detected object box.Part scores use a combination of arithmetic and geometric means.
  • Joint training is compared with object-only segmentation training for object segmentation performance.

H. Evaluation of open world detectors on zero-shot instance detection task

The paper evaluates open-world detectors on PACO’s zero-shot instance detection task using query vocabularies and query-specific inference, with results organized by query complexity and model configuration.

  • Detic uses all 5k PACO queries as a custom vocabulary and selects 2,000 detections per image for best results.
  • MDETR evaluates each of the 5k queries independently, merges detected boxes, and retains boxes with the top K query scores.
  • The open-world detector evaluation could not use ENB backbones because of a known issue.
  • The evaluation reports zero-shot instance detection results for L1, L2, L3, and all query levels.
  • Joint PACO training is evaluated on both PACO-LVIS and PACO-EGO4D using FPN models.PACO-EGO4D is described as the more challenging dataset.

I. Few-shot instance detection experiments

Few-shot instance detection uses separate detection and embedding towers to represent query and target boxes, then ranks target boxes by cosine similarity.

  • Query features are averaged across query images to form a single query feature vector.
  • Target boxes are ranked by cosine similarity to the query feature, and the top N boxes determine AR@N for N = 1, 5.
  • The few-shot model consists of a detector that predicts class-agnostic boxes and an embedding model that outputs representations for each box.
  • The embedding model replaces softmax training with ArcFace loss using instance IDs as supervision for richer instance representations.
  • The experiments include annotation examples and qualitative predictions for object boxes, part masks, and attribute outputs.
Loading 2301.01795v1…