Source-linked AI summary
Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations
Adel Ahmadyan, Liangkai Zhang, Jianing Wei, Artsiom Ablavatski, Matthias Grundmann
TL;DR
3D object understanding lacks large real-world datasets that match video or camera-stream inputs. Objectron addresses this gap with a large-scale object-centric video dataset, AR-derived metadata, pose annotations, and a 3D IoU evaluation method, then supplies detection baselines. The dataset is intended to support 3D detection and related multi-view applications, while annotation accuracy depends partly on camera-pose drift and rater consistency.
Problem
Large real-world datasets for 3D object understanding are scarce compared with 2D datasets, despite applications in augmented reality, robotics, autonomy, and image retrieval.
Method
The paper builds Objectron from short object-centric videos with AR session metadata, manually annotated 3D boxes, and an algorithm for computing 3D IoU for oriented boxes.
Results
The dataset contains 14,819 annotated video clips and 4M annotated images, and the paper provides baseline results for 3D object detection and viewpoint estimation.
Takeaways & Limitations
Multiple object views, temporally consistent boxes, metric-scale data, and dense annotations support 3D tracking, geometric understanding, view synthesis, shape reconstruction, and scale-aware methods.
Takeaways & Limitations
Annotation accuracy depends on camera-pose drift and rater accuracy; videos typically remain below 10 seconds, with empirically observed drift below 2% in length.
Abstract
from arXiv · showhide
3D object detection has recently become popular due to many applications in robotics, augmented reality, autonomy, and image retrieval. We introduce the Objectron dataset to advance the state of the art in 3D object detection and foster new research and applications, such as 3D object tracking, view synthesis, and improved 3D shape representation. The dataset contains object-centric short videos with pose annotations for nine categories and includes 4 million annotated images in 14,819 annotated videos. We also propose a new evaluation metric, 3D Intersection over Union, for 3D object detection. We demonstrate the usefulness of our dataset in 3D object detection tasks by providing baseline models trained on this dataset. Our dataset and evaluation source code are available online at http://www.objectron.dev
1. Introduction
Objectron addresses the scarcity of large real-world 3D datasets with object-centric videos that capture multiple views and rich pose information. Its annotations and data diversity support 3D detection and broader geometric applications.
- Objectron contains 14,819 annotated video clips and 4M annotated images from ten countries across five continents.
- Videos provide multiple views of the same objects, supporting multi-view understanding, view synthesis, and 3D shape reconstruction.
- Temporally consistent 3D bounding boxes throughout each video enable 3D tracking applications.
- Metric-scale translations and sizes, sparse point clouds, calibrated images, and camera parameters support scale recovery and sparse depth estimation.
- Each category includes hundreds of object instances collected across countries, locations, and lighting conditions.
- The authors propose accurate 3D IoU computation for oriented boxes and provide dataset-trained 3D detection baselines.
2. Previous Work
Previous 3D object datasets span controlled industrial scenes, image-based pose annotations, scene scans, and synthetic data. Objectron differs by emphasizing large-scale, high-resolution object-centric videos collected in the wild.
- Many established datasets use controlled environments, industrial objects, or relatively small collections of textured objects.
- Objectron provides larger-scale, high-resolution videos of common objects in the wild than these compared datasets.
- ObjectNet3D and Pascal3D+ provide image-based poses, but lack video, available camera intrinsics, and recoverable object scale.
- Scene datasets such as ScanNet contain indoor RGBD videos, while Objectron is object-centric and has an order of magnitude more samples than the cited scene datasets.
- Synthetic datasets offer training and benchmarking data, but their real-world generalization remains unknown.
3. Data Collection and Annotation
Objectron collects common objects in real-world settings using short videos captured from multiple viewpoints, then efficiently propagates verified 3D annotations across frames. AR tracking supplies metric information, while short sequences limit drift.
- Object Categories: Object categories span common rigid and non-rigid objects, including bikes, laptops, cups, bottles, books, cereal boxes, shoes, and chairs.The selection targets challenges involving non-rigid objects, rotational symmetry, text-rich textures, and real-time perception.
- Data Collection: Data collection records stationary objects while the camera moves around them, capturing multiple angles alongside AR camera poses, point clouds, and surface planes.Videos use high-end phone back cameras at 1440 × 1920 resolution and 30fps, with sequences kept around 10 seconds to reduce drift.
- Data Collection: The dataset spans 10 countries across five continents, with samples distributed uniformly in each region.Mobile phones enabled a rapid, geographically diverse collection campaign.
- Data Annotation: 3D boxes are annotated once in a video and populated to all frames using precomputed AR camera poses, reducing per-image annotation cost.Annotators adjust position, orientation, and scale so the projected box remains consistent across different views.
- Data Annotation: The annotation approach yields frame-level data while providing accurate metric bounding-box sizes through AR tracking.Its two stated benefits are annotating every frame from one video annotation and obtaining metric box sizes.
- Annotation Variance: Annotation accuracy depends on camera-pose drift and rater precision; observed drift was below 2%, and chair re-annotation showed low variance.For chairs, standard deviations were 4.6° for orientation, 1cm for translation, and 4cm for scale.
4. Objectron Dataset
The Objectron dataset provides annotated object-centric videos across nine categories, combining 3D object annotations with camera and environment metadata. Its annotations support pose and viewpoint analysis, with raters producing closely agreeing 3D boxes.
- Dataset Statistics: Objectron contains 17,095 object instances in 4M annotated images from 14,819 videos spanning nine categories.Videos show stationary objects from different camera viewpoints; some categories are non-rigid.
- Annotations: Each sample includes high-resolution images, camera poses, tracking point clouds, planar surfaces, and manually annotated 3D boxes.Boxes describe object orientation, translation, and size relative to the camera pose.
- Viewpoint Distribution: Viewpoint distributions are represented by azimuth polar plots and elevation plots for each object category.Some categories show bias toward front and top views.
- Annotations: The dataset derives 3D and projected 2D box keypoints, azimuth, elevation, and camera projection and view matrices from the pose annotations.These quantities are available for every video frame through the accompanying camera information.
5. Baseline Experiments and Evaluations
The paper defines evaluation for 3D detection through multiple metrics, emphasizing accurate 3D IoU for oriented boxes and reporting lightweight detection baselines. Results also reveal viewpoint-dependent performance differences.
- Evaluation metrics: The evaluation code computes average precision for 3D IoU, 2D projection, viewpoint, polar, and azimuth errors.
- 3D Intersection over Union: Accurate 3D IoU transforms both boxes into one box’s coordinate system, clips polygonal faces, and computes intersection volume from their convex hull.Rigid-body transformation preserves volume while simplifying clipping against an axis-aligned box.
- 3D Intersection over Union: For symmetric objects, the method rotates the estimated box around the symmetry axis and selects the orientation with maximum IoU.
- Baseline models: MobilePose baselines are trained separately for each category without pre-training or hyperparameter optimization, using the released evaluation code.Each model was trained for 100 epochs on eight V100 GPUs.
- Baseline models: The two baseline architectures are lightweight enough to achieve real-time performance on mobile devices.
- Baseline results: Elevation estimation is more accurate than azimuth estimation because elevation views are biased toward 45° while azimuth is uniformly distributed.The authors report particularly poor cup-rotation azimuth performance and suggest viewpoint-altering augmentation for generalization.
6. Details of the Objectron data format
The data format packages Objectron samples with videos, annotations, AR metadata, and model-evaluation resources. Tables and figures document baseline evaluations and category-specific statistics.
- Data format: Each sample includes a raw MOV video, AR metadata, and annotation results, with camera transformation, projection, and intrinsic matrices.
- Annotations: Object annotations store bounding-box orientation, translation, scale, 3D vertices in world and camera coordinates, and depth-aware 2D projections.
- Preprocessing: The dataset is also provided as TensorFlow tf.Example and tf.SequenceExample records for image and video training pipelines.Example pipelines target PyTorch, TensorFlow, and Jax.
- Evaluation resources: Table 2 evaluates different baseline models, while Figures 10 and 11 show MobilePose and two-stage network evaluations.
7. Conclusion
The paper concludes by presenting Objectron as a large-scale, in-the-wild pose dataset built with scalable on-device AR annotation. It positions the release as infrastructure for broader 3D understanding research and applications.
- Conclusion: Objectron contains 14,819 short videos with object pose annotations and is introduced as a large-scale object-centric dataset.
- Conclusion: The data collection and annotation framework is efficient and scalable because it is based on on-device AR libraries.
- Conclusion: The paper presents a two-stage 3D object detection model trained on Objectron as a baseline.
- Conclusion: The release is intended to support research in 3D geometry understanding, video models, object retrieval, view synthesis, and 3D reconstruction.