PaperlayerView exact revision on arXiv

Source-linked AI summary

ImageNet3D: Towards General-Purpose Object-Level 3D Understanding

Wufei Ma, Guanning Zeng, Guofeng Zhang, Qihao Liu, Letian Zhang, Adam Kortylewski, Yaoyao Liu, Alan Yuille

arXiv:2406.09613v1cs.CV

TL;DR

Object-level 3D understanding requires models to infer 2D and 3D information for arbitrary rigid objects, but existing datasets are limited in category coverage or annotation quality. ImageNet3D adds annotations across 200 categories and considers new 3D-awareness and open-vocabulary pose tasks; experiments report that it supports general-purpose models across a wide range of rigid categories. The dataset may be less suitable for tasks requiring object co-occurrences because its images are mostly object-centric with one or two instances.

  • Problem

    Existing object-level 3D datasets are limited in category coverage or annotation quality, leaving unified models for common rigid objects in natural images understudied.

    Source: §2 · p. 3

  • Method

    ImageNet3D adds 2D boxes, 6D poses, cross-category alignment, and captions interleaved with 3D information for 200 categories, and evaluates standard and new 3D tasks.

    Source: §1 · p. 2

  • Results

    Experiments show ImageNet3D supports general-purpose models that infer 3D information across a wide range of rigid categories.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    ImageNet3D provides annotations for studying object-level 3D awareness and open-vocabulary pose estimation, as well as integrating 3D understanding with large language models.

    Source: §1 · p. 2

    Because most images are object-centric with one or two instances, ImageNet3D may not suit 3D detection or tasks requiring object co-occurrences.

    Source: B Limitations · p. 15

Abstract

from arXiv · show

A vision model with general-purpose object-level 3D understanding should be capable of inferring both 2D (e.g., class name and bounding box) and 3D information (e.g., 3D location and 3D viewpoint) for arbitrary rigid objects in natural images. This is a challenging task, as it involves inferring 3D information from 2D signals and most importantly, generalizing to rigid objects from unseen categories. However, existing datasets with object-level 3D annotations are often limited by the number of categories or the quality of annotations. Models developed on these datasets become specialists for certain categories or domains, and fail to generalize. In this work, we present ImageNet3D, a large dataset for general-purpose object-level 3D understanding. ImageNet3D augments 200 categories from the ImageNet dataset with 2D bounding box, 3D pose, 3D location annotations, and image captions interleaved with 3D information. With the new annotations available in ImageNet3D, we could (i) analyze the object-level 3D awareness of visual foundation models, and (ii) study and develop general-purpose models that infer both 2D and 3D information for arbitrary rigid objects in natural images, and (iii) integrate unified 3D models with large language models for 3D-related reasoning.. We consider two new tasks, probing of object-level 3D awareness and open vocabulary pose estimation, besides standard classification and pose estimation. Experimental results on ImageNet3D demonstrate the potential of our dataset in building vision models with stronger general-purpose object-level 3D understanding.

1 Introduction

General-purpose object-level 3D understanding requires inferring both 2D and 3D properties of arbitrary rigid objects, but existing datasets and models have limited category or domain coverage. ImageNet3D addresses this gap with annotations and designs intended to support unified-model research, new evaluations, and 3D reasoning.

1 Introduction

  • Existing 3D datasets cover few categories or specific domains, and models developed for them can fail to generalize to novel categories.The need for unified models across common rigid objects in natural images remains understudied.Source: §1 · p. 1
  • ImageNet3D extends 200 ImageNet21k categories with 2D bounding boxes and 6D poses for more than 86,000 objects, plus cross-category alignment and captions interleaved with 3D information.The dataset targets arbitrary rigid objects in natural images.Source: §1 · p. 2, Figure 1 · p. 2
  • ImageNet3D aligns canonical poses across 200 categories using semantic parts, shapes, and common knowledge to support joint learning and generalization to unseen categories.The authors describe cross-category alignment as crucial for models to use information shared across categories.Source: §1 · p. 2
  • ImageNet3D captions interleave image descriptions with 3D information, providing assets for integrating unified 3D models with large language models.The captions are generated with a GPT-assisted approach.Source: §1 · p. 2, Figure 1 · p. 2
  • The paper considers probing object-level 3D awareness and open-vocabulary pose estimation alongside standard classification and pose estimation.Experiments report that ImageNet3D supports general-purpose models that infer 3D information across a wide range of rigid categories, while baseline results expose limitations and challenges.Source: §1 · p. 2

2 Related Works

Prior 3D datasets and methods often cover few categories or specific domains, while annotation quality can also constrain unified-model development. ImageNet3D broadens category and instance coverage and adds cross-category alignment and natural captions, alongside work on category-level pose estimation and visual-model 3D awareness.

Datasets with 3D annotations

  • ImageNet3D expands category and instance coverage, improves annotation quality, and adds cross-category alignment and natural captions interleaved with 3D information.The dataset is positioned to support unified models and studies of object-level 3D awareness and open-vocabulary pose estimation.Source: §2 · p. 3, Table 1 · p. 3

Category-level pose estimation

  • Earlier category-level pose methods explored keypoints [31] and 3D compositional models [32] [33], but covered few categories; ImageNet3D also considers pose estimation for similar novel categories.Parallel work [34] [35] discussed open-vocabulary pose estimation, while [35] used synthetic data rendered with photorealistic CAD models.Source: §2 · p. 3

3D awareness of visual foundation Models

  • Banani et al. [22] probed visual models’ 3D awareness through surface reconstruction and multi-view consistency, while ImageNet3D studies a more comprehensive range of 3D structure.The passage identifies the absence of large 3D-annotated datasets as the reason for the earlier work’s limited scope.Source: §2 · p. 3

3 ImageNet3D Dataset

ImageNet3D is built to support research on general-purpose object-level 3D understanding, with annotations spanning 200 categories and more than 86,000 objects. Its construction combines 2D and 3D annotation, aligned canonical poses across categories, and captions interleaved with 3D information.

3.1 Dataset Construction

  • ImageNet3D extends 200 categories with 6D pose annotations for more than 86,000 objects, using ImageNet21k images, machine-assisted boxes, category CAD models, and human pose annotation.The 200 categories were selected for rigidity, well-defined shapes with variance, and sufficient image availability; annotators also label scene density and object visual quality.Source: §3 · p. 3, §3.1 · pp. 3–4

3.2 Cross-Category 3D Alignment

  • Canonical poses are manually aligned across all 200 categories using shared semantic parts, similar shapes, and common-knowledge front directions.The alignment is intended to let models exploit similarities across categories and infer viewpoints for novel categories from semantic parts seen during training.Source: §3.2 · pp. 4–5, Figure 2 · p. 4

3.3 Natural Captions with 3D Information

  • ImageNet3D captions describe image content, object appearances and locations, and mutual relations, while <pose6d> links annotated objects to their 2D and 3D annotations.The captions are generated through a GPT-assisted process that supplies annotations in prompts and uses GPT-4v to produce coherent captions.Source: §3.3 · p. 5

4 Tasks

ImageNet3D supports research on general-purpose object-level 3D understanding through three tasks: probing 3D awareness, open-vocabulary pose estimation, and joint image classification with category-level pose estimation.

4 Tasks

  • ImageNet3D supports three tasks: probing object-level 3D awareness, open-vocabulary pose estimation, and joint image classification with category-level pose estimation.These tasks are presented alongside the dataset’s goal of advancing evaluation and research in general-purpose object-level 3D understanding.Source: §4 · p. 5

4.1 Linear Probing of Object-Level 3D Awareness

  • Frozen visual features are linearly probed on 3D viewpoint classification to evaluate object-level 3D awareness.Three linear classifiers predict the parameters encoding viewpoint, with hyperparameters selected by grid search.Source: §4.1 · p. 6

4.2 Open-Vocabulary Pose Estimation

  • Open-vocabulary pose estimation tests generalization to 137 categories unseen during training, using 3D annotations from 63 common categories.At test time, models also have access to annotated category-level captions; additional 2D pretraining data is permitted.Source: §4.2 · p. 7

4.3 Joint Image Classification and Category-Level Pose Estimation

  • Joint classification and category-level pose estimation counts a prediction as correct only when both its class label and pose satisfy the criteria.The task uses about 50% of each category’s samples for training and the remainder for validation, enabling zero-shot, few-shot, and fully supervised settings.Source: §4.3 · p. 7

5 Experimental Results

The experiments evaluate object-level 3D awareness, open-vocabulary pose estimation, and joint classification with category-level pose estimation on ImageNet3D. Results show measurable 3D awareness in vision foundation models, limited generalization to novel categories, and general-purpose 3D inference across rigid categories.

5 Experimental Results

  • The baseline experiments cover linear probing of 3D awareness, open-vocabulary pose estimation, and joint image classification with category-level pose estimation.All reported experiments use data from 189 ImageNet3D categories.Source: §5 · p. 7

5.1 Linear Probing of Object-Level 3D Awareness

  • DINO v2 outperforms the other tested approaches in object-level 3D awareness, followed by MAE, DINO, and MiDaS.The linear probe uses frozen class-embedding features from general-purpose vision models.Source: §5.1 · p. 8, Table 2 · p. 8

5.2 Open-Vocabulary Pose Estimation

  • Category-agnostic models generalize to novel categories with reasonable performance, but current 3D models remain limited compared with models trained on annotations from those categories.The evaluation reports π/6 pose estimation accuracy on novel categories unseen during training.Source: §5.2 · p. 8, Table 3 · p. 9

5.3 Image Classification and Category-Level Pose Estimation

  • ImageNet3D supports general-purpose models that infer 3D information across rigid categories, while 3D-aware classification accuracy degrades as the number of categories scales up.The experiments use 189 categories after annotations for some categories were returned for refinement.Source: §5 · p. 7, Table 4 · p. 9, §5.3 · p. 9

6 Conclusion

ImageNet3D expands object-level 3D annotations across rigid categories and instances, adds cross-category alignment and new annotation types, and supports both established and new evaluation tasks. Experiments show that models developed with the dataset can infer 3D information across a wide range of rigid categories.

6 Conclusion

  • ImageNet3D expands the number of rigid categories and object instances relative to previous 3D-annotation datasets, and improves annotation quality through cross-category 3D alignment.It also provides object visual qualities and captions interleaved with 3D information.Source: §6 · p. 9
  • The paper reports baseline results for standard 3D tasks and the new tasks of probing object-level 3D awareness and open-vocabulary pose estimation.Source: §6 · p. 9
  • Experiments show that ImageNet3D can support general-purpose models that infer 3D information for a wide range of rigid categories.The paper also identifies limitations of existing 3D models and discusses challenges for future work.Source: §6 · p. 9

A.1 Dataset Details

ImageNet3D selects common rigid categories with well-defined shapes and varied viewpoints, and resolves viewpoint ambiguity through category-specific common-viewpoint conventions. The appendix also describes annotation guidance, caption generation, and ethics procedures.

A.1 Dataset Details

  • The 200 ImageNet21k categories were selected as common rigid object classes with well-defined shapes and varied 3D viewpoints, excluding narrow subsets and unsuitable categories.Animals, clothes, food, chimes, and categories often pictured from a single viewpoint are among the exclusions described.Source: §A.1 · p. 14
  • To resolve viewpoint ambiguity, the dataset assigns a common viewpoint to categories such as tables and bottles, allowing standard pose-estimation metrics during evaluation.The authors follow prior work for this convention.Source: §A.1 · p. 14

A.2 Annotator Guidelines

  • Annotator guidelines and tutorials explain annotation parameters, use of the web app, and handling of boundary cases.Source: §A.2 · p. 14

A.3 Caption Generation

  • Image captions interleave 3D annotations after an object's first mention, while category-level captions describe shape, part structure, and human interactions.GPT-4v generates natural-image captions using 3D annotations and object bounding boxes as context; category-level captions are manually annotated.Source: §A.3 · p. 14

A.4 Ethics and Institutional Review Board (IRB)

  • The work received IRB approval, disclosed potential risks to annotators, and paid annotators fairly according to institutional requirements.Source: §A.4 · pp. 14–15

B Limitations

ImageNet21k images are mostly object-centric, limiting the dataset's suitability for detection and co-occurrence tasks. The appendix describes classification-based and 3D compositional baselines, including NMM-Sphere and LLaVA-pose.

B Limitations

  • Because most ImageNet21k images contain only one or two object instances, ImageNet3D may not suit 3D detection or tasks requiring object co-occurrences.The authors note that compositional models trained on object-centric data have been found to generalize to real images with multiple objects or partial occlusion [32] [33].Source: B Limitations · p. 15

C.1 Baseline Models

  • Classification-based baselines predict pose by learning three classifiers over 40 bins per pose parameter, using cross-entropy loss.Source: §C.1 · p. 15
  • NMM-Sphere uses a unified sphere mesh shared across categories and can support joint classification and pose estimation or open-vocabulary pose estimation.The joint setting uses a class-contrastive loss, while the open-vocabulary setting learns category-agnostic features.Source: §C.1 · p. 15
  • LLaVA-pose extends LLaVA with a <pose> token decoded by a classifier head to predict pose.Source: §C.1 · p. 15

C.2 Training Details

Training details use restrained augmentation and set task-specific baseline procedures. Linear probing selects settings by validation accuracy, while pose-estimation training uses fixed schedules and equal loss weights for the joint task.

C.2 Training Details

  • Baseline models across all three tasks use only random horizontal flips, with more complex augmentation left for future study.Source: §C.2 · p. 15
  • Linear probing grid-searches learning rates, pooling strategies, and backbone blocks, then reports the best validation accuracy.The selected settings determine which frozen features and pooling strategy are evaluated.Source: §C.2 · p. 15
  • Open-vocabulary classification-based methods train for 120 epochs with batch size 64 and SGD initialized at learning rate 0.01.Source: §C.2 · p. 15
  • Joint classification and pose estimation reuse the open-vocabulary training strategy and assign both losses weight 1.0.Source: §C.2 · p. 15

D Data and Code Release

ImageNet3D is released under CC BY-NC 4.0, with users also bound by the original ImageNet terms. The release notes flag potentially harmful content and identifiable faces in some images, and direct users to ImageNet21k guidelines.

D.1 License

  • The dataset and its annotations are released under CC BY-NC 4.0, and users must also follow the original ImageNet access terms and license.Source: §D.1 · p. 15

D.2 Risks and Concerns

  • A very small number of images may be offensive, insulting, threatening, or anxiety-inducing; these images come from ImageNet21k, whose guidelines users should follow.Source: §D.2 · p. 16
  • Some images may contain identifiable faces; these images also come from ImageNet21k, so its guidelines apply.Source: §D.2 · p. 16

D.3 ImageNet3D Dataset and Code

  • The release lists the dataset datasheet, raw data, Croissant metadata, experiment and web-app code, and annotator materials.The listed materials include an annotator tutorial and guidelines.Source: §D.3 · p. 16

E Human Evaluation of Annotation Quality

Human evaluators generally favored ImageNet3D’s 3D annotations over ObjectNet3D’s. The paper also reports object-level awareness and pose-estimation baselines, with differing scaling behavior across models and limited generalization to novel categories.

E Human Evaluation of Annotation Quality

  • 73.25% of evaluated samples favored ImageNet3D annotations over ObjectNet3D, with ImageNet3D generally favored in all 16 tested categories.The comparison sampled 50 images per category and asked evaluators to judge alignment of viewpoint and object location.Source: E Human Evaluation of Annotation Quality · p. 16, Table 5 · p. 17

F.1 Linear Probing of Object-Level 3D Awareness

  • DINO v2 and DeiT do not scale well as model parameters increase, while MAE outperforms DINO v2 at large and huge sizes.Figure 7 presents scaling properties across backbones; Table 6 reports π/6 pose accuracy, including DINO v2 as best among tested foundation models.Source: §F.1 · p. 16, Figure 7 · p. 18, Table 6 · p. 18

F.3 Joint Image Classification and Category-Level Pose Estimation

  • At comparable sizes near ViT-B, Swin Transformer performs better, but ResNet outperforms it as model sizes scale up.Figure 8 shows scaling for joint classification and category-level pose estimation; Table 8 reports 3D-aware classification accuracy at π/6 and notes limitations when scaling to many categories.Source: §F.3 · p. 16, Table 8 · p. 19, Figure 8 · p. 20
  • GPT-4v captions interleave natural-language image descriptions with 3D information, using provided 3D annotations and object names as context.The captions are presented as a way to integrate general-purpose 3D models with large language models.Source: Figure 6 · p. 17

F.2 Open-Vocabulary Pose Estimation

  • Table 7 reports π/6 pose accuracy on novel categories, and its caption characterizes category-agnostic features as enabling limited generalization.Source: §F.2 · p. 16, Table 7 · p. 19
Loading 2406.09613v1…