Source-linked AI summary
ABO: Dataset and Benchmarks for Real-World 3D Object Understanding
Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F. Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthieu Guillaumin, Jitendra Malik
TL;DR
ABO addresses the shortage of large-scale datasets that pair diverse real-world images with realistic 3D objects and materials. It constructs a product-derived dataset and benchmarks reconstruction, material estimation, and cross-domain retrieval. Across these tasks, the authors report challenging evaluations, improved SV-BRDF disentanglement with multiple views, and retrieval performance below common saturated benchmarks.
Problem
Real-world 3D understanding lacks large-scale datasets combining diverse images, realistic geometries, and physically-based materials across household objects.
Method
ABO combines product catalog images, metadata, artist-designed 3D meshes, and synthetic renders to benchmark reconstruction, material estimation, and cross-domain multi-view retrieval.
Results
ABO provides challenging evaluations: multiple views improve SV-BRDF disentanglement, while retrieval remains substantially below existing common benchmarks.
Takeaways & Limitations
ABO supports evaluation of real-world 3D understanding under complex geometries, realistic materials, novel viewpoints, and varied scenes.
Takeaways & Limitations
Prior exact image-to-model datasets remain small and category-limited, while reconstructed datasets require extensive manual effort and controlled imaging.
Abstract
from arXiv · showhide
We introduce Amazon Berkeley Objects (ABO), a new large-scale dataset designed to help bridge the gap between real and virtual 3D worlds. ABO contains product catalog images, metadata, and artist-created 3D models with complex geometries and physically-based materials that correspond to real, household objects. We derive challenging benchmarks that exploit the unique properties of ABO and measure the current limits of the state-of-the-art on three open problems for real-world 3D object understanding: single-view 3D reconstruction, material estimation, and cross-domain multi-view object retrieval.
1 UC Berkeley, 2 Amazon, 3 BITS Pilani
ABO combines product images with realistic, high-resolution, physically-based 3D models of household objects to benchmark real-world object understanding.
- ABO pairs product images with realistic, high-resolution, physically-based 3D models of household objects.
1. Introduction
Existing real-world 3D datasets are limited by scale, category coverage, approximate alignment, or unrealistic geometry and materials. ABO addresses these gaps with diverse product imagery, metadata, realistic 3D models, and benchmarks spanning reconstruction, material estimation, and retrieval.
- Large-scale real-world 3D annotation is difficult, while synthetic CAD datasets often contain low-quality or untextured models absent from the real world.
- Existing image-aligned datasets remain small, category-limited, and largely untextured, restricting their use beyond shape or pose tasks.IKEA and Pix3D contain 90 and 395 unique 3D models, respectively, and Pix3D covers only 9 object categories.
- Reconstructed real-world datasets require substantial manual effort, use controlled imaging settings, lack in-context images, and assume Lambertian surface reflectance.
- ABO contains 147,702 product listings, 398,212 catalog images, metadata for up to 18 attributes, 8,222 turntable-image products, and 7,953 products with artist-designed 3D meshes.Its meshes have complex geometries and high-resolution physically-based materials for photorealistic rendering.
- ABO benchmarks single-view shape reconstruction, material estimation from single or multiple views, and image-based multi-view object retrieval.The benchmarks target synthetic-to-real transfer, spatially varying BRDF estimation, and viewpoint- and scene-robust metric learning.
2. Related Work
Prior 3D datasets trade off scale, realism, category diversity, or exact image–model alignment. ABO addresses these gaps with realistic artist-created shapes and materials, and benchmarks reconstruction, material estimation, and multi-view retrieval across more diverse settings.
- Material Estimation: Existing material datasets often use simplistic or procedural shapes, whereas ABO provides artist-created real-object shapes and spatially varying BRDFs for photorealistic rendering.
- 3D Object Datasets: ABO is the only compared dataset combining real images, full 3D meshes, PBR materials, and substantially broader category diversity.
- 2D/3D Image Retrieval: ABO’s multi-view retrieval benchmark uses hundreds of diverse categories and 3D structure to test viewpoint- and scene-robust representations beyond near-saturated benchmarks.
3. The ABO Dataset
ABO combines worldwide product listings, catalog imagery, metadata, artist-created 3D models, and automatically estimated image poses. Its dataset design supports reconstruction evaluation, material rendering, and real-world 3D understanding benchmarks.
- Dataset Properties: ABO contains 147,702 product listings from 576 product types, with structured metadata derived from publicly available product pages.
- 3D Models: The dataset includes 7,953 artist-created glTF 2.0 models in canonical coordinates, real-world scale, and WordNet-linked category annotations.
- Catalog Image Pose Annotations: ABO provides 6-DOF pose annotations for 6,334 catalog images using an automated pipeline based on masks, differentiable rendering, and the known 3D model.
- Catalog Image Pose Annotations: The pose pipeline minimizes silhouette loss over rotation R ∈ SO(3) and translation T ∈ R^3, followed by a final human verification step.
- Material Estimation Dataset: For material estimation, ABO renders 512x512 images from 91 camera positions under randomized indoor HDRI lighting using Disney base color, metallic, and roughness parameters.
4. Experiments
ABO experiments evaluate ShapeNet-trained reconstruction, SV-BRDF material estimation, and cross-domain retrieval on realistic objects and images. The results expose substantial domain and viewpoint challenges while showing benefits from multi-view material inputs.
- 4.1. Evaluating Single-View 3D Reconstruction: ShapeNet-trained reconstruction methods are evaluated on 4,170 ABO models across six categories shared with ShapeNet.The experiment uses 30 rendered viewpoints per object, with azimuth and elevation sampled on a unit sphere while excluding uncommon bottom views.
- 4.1. Evaluating Single-View 3D Reconstruction: Mesh R-CNN performs best on Chamfer Distance, while Occupancy Networks performs best on Absolute Normal Consistency across ABO and ShapeNet evaluations.The comparison includes voxel, spherical-map, implicit-function, and mesh representations, with both view-space and canonical-space predictions.
- 4.1. Evaluating Single-View 3D Reconstruction: A large performance gap separates ShapeNet and ABO predictions, indicating that real-world ABO shapes and textures are out of distribution for ShapeNet-trained models.The lamp category shows a particularly large drop, likely because thin structures are difficult to reconstruct.
- 4.2. Material Prediction: The U-Net material baseline uses a ResNet-34 encoder and separate decoder heads to estimate base color, roughness, metallicness, surface normals, and rendering properties.For multi-view prediction, neighboring images are projected into alignment with the reference view using depth maps.
- 4.2. Material Prediction: The multi-view network outperforms the single-view network on base color, roughness, metallicness, and surface-normal prediction, especially for view-dependent specular properties.Removing 3D-structure-based projection still improves roughness and metallicness over the single-view network, but projection improves all parameters.
- 4.2. Material Prediction: The multi-view network makes reasonable material predictions on real catalog images despite differences in lighting, background, and synthetic-to-real domain.The inferred material parameters are used to relight test objects from catalog images with pose annotations.
- 4.3. Multi-View Cross-Domain Object Retrieval: 5% Recall@1 is achieved by the ImageNet-trained ResNet-50 baseline, while NormSoftmax, ProxyNCA, and Contrastive reach approximately 29% on the ABO retrieval benchmark.These methods outperform Multi-similarity, NTXent, and TripletMargin, which reach approximately 23%.
- 4.3. Multi-View Cross-Domain Object Retrieval: Retrieval performance degrades rapidly beyond azimuth |θ| > 75° and elevation φ > 50° for all evaluated methods.The benchmark uses rendered queries and catalog images to test robustness across product viewpoints.
5. Conclusion
ABO is introduced as a dataset for bridging real and synthetic 3D worlds through realistic objects, images, metadata, and artist-created models. Its benchmarks reveal weak reconstruction transfer, improved SV-BRDF estimation from multiple views, and challenging viewpoint-sensitive retrieval.
- 5. Conclusion: ABO provides real-world-derived 3D models that challenge both view-space and canonical-space methods trained on ShapeNet, even within shared training categories.The conclusion identifies ABO as a test set for evaluating transfer from synthetic to real-world objects.
- 5. Conclusion: Multiple views produce more accurate disentanglement of SV-BRDF properties for complex real-world geometries.This material-estimation task is enabled by the nature of ABO’s 3D dataset.
- 5. Conclusion: ABO combines product images with synthetic renders to create a challenging retrieval task that evaluates deep metric learning across query azimuth and elevation.The benchmark addresses diversity and structural limitations of existing datasets described as close to performance saturation.
- 5. Conclusion: ABO’s text annotations, non-rigid products, household objects, weights, and dimensions support possible language, vision, robotics, and simulation tasks beyond those evaluated here.The conclusion notes examples including style, pattern, caption, keyword, and manipulation-related applications.
A. Dataset Properties
ABO’s metadata visualization organizes products by selected attributes, showing categorical labels as rows and continuous values as ordered bins.
- Metadata Visualization: Figure 9 groups products by unit count and weight, using rows for categorical labels or continuous-valued bins.Within each continuous-valued row, products are ordered from lowest to highest.
B. Dataset Organization
ABO’s data subsets are assigned to training and testing roles for benchmarks in 3D reconstruction, material estimation, and multi-view retrieval.
- Dataset Organization: Table 6 specifies how each ABO data subset is used at train and test time across 3D reconstruction, material estimation, and multi-view retrieval.No-BG Renders are white-background ABO object renders, while BG Renders refer to the Material Estimation Dataset.
C. 6DOF Pose Optimization
Instance masks come from pretrained Mask R-CNN and PointRend models, while pose optimization searches over randomized rotations and optimized translations before selecting the lowest-loss pose.
- Instance masks are generated with Mask R-CNN trained on LVIS and PointRend trained on COCO.Predicted masks from all categories are retained when confidence exceeds 0.1.
- 24 randomized pose-optimization runs optimize R and T for 1,000 Adam steps at a 1e-2 learning rate.Rotations use symmetric orthogonalization, and the lowest-loss result is selected and human-validated.
D. Single-View 3D Reconstruction Evaluation
The evaluation aligns predicted and ground-truth meshes differently in view-space and canonical-space before measuring reconstruction quality on ABO objects and examining qualitative failures.
- View-Space Evaluations: View-space evaluation resolves depth ambiguity by aligning predicted and ground-truth meshes using known camera extrinsics and Chamfer-distance minimization.Meshes are independently normalized by average vertex depth and scaled using the ground-truth bounding-box convention.
- Canonical-Space Evaluations: Canonical-space evaluation applies one manually set rotation alignment, then searches scale and translation to minimize Chamfer distance.Meshes are mean-centered and independently rescaled before a 34-grid search over candidate scale and translation.
- Reconstruction quality is reported using Chamfer distance and absolute normal consistency averaged across categories.
- Additional Qualitative Examples: Both GenRe and Mesh R-CNN generally fail on thin structures, with GenRe omitting them and Mesh R-CNN producing convex-hull-like geometry.The qualitative comparison covers ABO objects overlapping ShapeNet categories and categories absent from ShapeNet.
- Quantitative Evaluation on Test Split: The reconstruction benchmark evaluates ShapeNet-category ABO models to measure how ShapeNet-trained methods transfer to ABO.An 80%/10%/10% train/validation/test split is also released for future ABO-based training.
E. Material Estimation
The material-estimation benchmark uses curated PBR data and compares single-view with multi-view prediction, where camera geometry establishes correspondences and UV aggregation produces full texture maps.
- Dataset Curation: The material-estimation dataset contains 7,679 non-transparent models split into 6,897 training and 782 test models.Ten of 108 HDRI environment maps are reserved exclusively for testing generalization to new lighting.
- Network Details: The multi-view network uses camera poses to establish pixel-level correspondences between viewpoints and fills occluded pixels from the reference view.Correspondences use camera intrinsics, relative rotation and translation, and pixel depth, followed by a depth-based occlusion test.
- Full Texture Map Reconstruction: Per-view material predictions are back-projected into UV space and processed by a learned network for aggregation and smoothing.This pipeline generates a full textured PBR model.
- Network Details: The single-view and multi-view material-estimation architectures are presented as encoder-decoder baselines.Figure 10 shows the single-view and multi-view network configurations.
F. Multi-View Cross-Domain Object Retrieval
The ABO-MVR benchmark evaluates retrieval between rendered and catalog imagery across product groups, using balanced deep metric-learning training and multiple ranking metrics; rendered-query retrieval is substantially harder.
- Dataset Curation: The dataset focuses retrieval on rigid products, groups near-duplicate listings into shared instances, and organizes product lines into product groups.Near-duplicates are treated as correct matches of a single instance, while groups may share design details, materials, patterns, or imagery.
- Dataset Curation Statistics: The test set contains 836 instances with 4,313 catalog target images and eight rendered query views per environment map.Hierarchical Union-Find produced 29,988 groups and 50,756 instances overall.
- Implementation Details: Retrieval models use an unfrozen ImageNet-pretrained ResNet-50, a 128D embedding module, and class- and domain-balanced sampling.The backbone produces a 2048D vector before LayerNorm and linear projection.
- Additional Metrics: The benchmark reports Recall@k, MAP, MAP@R, and R-Precision to assess retrieval accuracy and ranking quality.The additional metrics reward correct results appearing earlier and maximizing correct results among top ranks.
- Quantitative Results: Rendered queries retrieve substantially worse than catalog queries, highlighting the difficulty of cross-domain retrieval.Rendered-image queries search catalog galleries containing both train and test classes, while catalog queries use the same gallery construction.
- Qualitative Results: Qualitative examples compare Contrastive, NormSoftmax, and ProxyNCA on low-, mid-, and high-elevation queries.Top-8 results are marked correct or incorrect, allowing viewpoint-dependent successes and failures to be inspected.