Source-linked AI summary
ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer
Ruohan Gao, Zilin Si, Yen-Yu Chang, Samuel Clarke, Jeannette Bohg, Li Fei-Fei, Wenzhen Yuan, Jiajun Wu
TL;DR
Prior object models often emphasize 2D images or geometry and provide limited multisensory representation. ObjectFolder 2.0 encodes visual, acoustic, and tactile data for 1,000 objects; models trained on them transfer to three real-world tasks.
Problem
Prior object modeling often focuses on 2D images or geometry, with low-quality textures and limited representation of physical properties beyond vision.
Source: §1 · p. 1Method
ObjectFolder 2.0 represents 1,000 real-world objects as implicit neural representations encoding visual, acoustic, and tactile data.
Source: §1 · p. 2Results
Models learned with virtualized objects transfer successfully to real-world counterparts across three challenging tasks; the dataset is 10 times larger and orders of magnitude faster to render than existing work.
Source: §7 · p. 8Takeaways & Limitations
The dataset offers a potential testbed for multisensory learning, with applications including robotic grasping and on-the-fly multisensory data generation.
Source: §1 · p. 2The dataset models rigid objects with one homogeneous material, while real objects may have non-rigid parts and multiple materials, and transfer also faces varied lighting, noise, and reverberation.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Objects play a crucial role in our everyday activities. Though multisensory object-centric learning has shown great potential lately, the modeling of objects in prior work is rather unrealistic. ObjectFolder 1.0 is a recent dataset that introduces 100 virtualized objects with visual, acoustic, and tactile sensory data. However, the dataset is small in scale and the multisensory data is of limited quality, hampering generalization to real-world scenarios. We present ObjectFolder 2.0, a large-scale, multisensory dataset of common household objects in the form of implicit neural representations that significantly enhances ObjectFolder 1.0 in three aspects. First, our dataset is 10 times larger in the amount of objects and orders of magnitude faster in rendering time. Second, we significantly improve the multisensory rendering quality for all three modalities. Third, we show that models learned from virtual objects in our dataset successfully transfer to their real-world counterparts in three challenging tasks: object scale estimation, contact localization, and shape reconstruction. ObjectFolder 2.0 offers a new path and testbed for multisensory learning in computer vision and robotics. The dataset is available at https://github.com/rhgao/ObjectFolder.
1. Introduction
ObjectFolder 2.0 addresses the limited realism and multisensory coverage of prior object models with a larger implicit-representation dataset. Models trained on its virtualized objects transfer to three real-world tasks.
1. Introduction
- Prior object models often focus on 2D images or geometry, with low-quality textures and limited representation of physical properties beyond vision.Examples include 2D recognition and locating, or 3D CAD models focused purely on geometry.Source: §1 · p. 1
- The approach uses scans of real objects to encode texture, material, and shape, simulate visual, acoustic, and tactile data, and represent them with Object File.The resulting sensory data can be rendered based on object extrinsics.Source: §1 · pp. 1–2, Figure 1 · p. 1
- ObjectFolder 2.0 contains 1,000 objects, improves visual, acoustic, and tactile rendering, and transfers to three real-world tasks.It is 10 times larger than existing work and renders orders of magnitude faster.Source: §1 · p. 2
- The dataset is intended to support multisensory learning, robotic grasping, and applications needing on-the-fly multisensory data, including on-policy reinforcement learning.Source: §1 · p. 2
2. Related Work
Prior object datasets variously emphasize 2D images, geometry, or vision alone, while multisensory learning work often targets particular tasks. ObjectFolder 2.0 instead offers implicit object models with visual, acoustic, and tactile data as a potential shared testbed.
Object datasets
- Existing object datasets include 2D image collections, geometry-focused CAD models, limited or approximate 2D–3D alignments, and small real-object collections; ABO has over 8K visual-only models.Source: §2 · p. 2
- ObjectFolder 2.0 contains 1,000 implicit 3D objects encoding visual, acoustic, and tactile data, and is 100 times faster to render than ObjectFolder 1.0.Unlike ObjectFolder 1.0, it also demonstrates generalization to real-world counterparts.Source: §2 · p. 2
Implicit neural representations
- The paper uses MLPs for object-centric sensory data and introduces representations that encode object intrinsics more realistically and flexibly while enabling real-time rendering across modalities.Source: §2 · p. 2
Multisensory learning
- ObjectFolder 2.0 is proposed as a potential testbed for multisensory learning across all three modalities, rather than a dataset for one particular task or modality combination.Source: §2 · p. 2
3. A Large Repository of Diverse Objects
The repository comprises 1,000 implicit 3D object models assembled from prior collections and filtered for material properties. It includes diverse household items and retains objects judged approximately homogeneous in material.
3. A Large Repository of Diverse Objects
- The 1,000-object repository combines all 100 objects from ObjectFolder 1.0, 855 objects from ABO, and 45 polycarbonate objects from Google Scanned Objects.Source: §3 · p. 3
- ABO objects were filtered to seven material types, metadata-checked against product images, and retained when their material was approximately homogeneous.The selected materials are ceramic, glass, wood, plastic, iron, polycarbonate, and steel; the filtering supports acoustic simulation.Source: §3 · p. 3
- The collection includes household items across diverse categories, such as wood desks, ceramic bowls, plastic toys, steel forks, and glass mirrors.Source: §3 · p. 3
4. Improved Multisensory Simulation and Implicit Representations
This section presents Object File implicit representations and modality-specific pipelines for visual, acoustic, and tactile rendering. It reports improved multisensory quality and real-time rendering compared with ObjectFolder 1.0.
4. Improved Multisensory Simulation and Implicit Representations
- Object File encodes each object's visual, acoustic, and tactile data as continuous functions of sensing conditions, allowing arbitrary-resolution sampling and memory use independent of those conditions.The object representation is a neural network that encodes the object's complete multisensory profile.Source: §4 · p. 3
4.1. Vision
- KiloOSF partitions each object into spatial cells represented by small MLPs, using distillation and ray-skipping techniques to accelerate rendering while improving visual quality.The approach renders images that match ground truth well and makes VisionNet inference 60 times faster than OBJECTFOLDER 1.0.Source: §4.1 · p. 4
4.2. Audio
- AudioNet uses higher-order tetrahedral modal analysis and predicts contact-dependent mode gains, then combines them with object-intrinsic frequencies and damping to synthesize waveforms.Predicting location-dependent signal components rather than full spectrograms improves audio rendering quality.Source: §4.2 · pp. 4–5
4.3. Touch
- TouchNet maps contact location, orientation, gel depth, and deformation-map pixel to deformation values, then uses Taxim to render tactile RGB images.It supports contact rotations within ±15° and pressing depths of 0.5–2 mm, and its optical mapping can be calibrated to real vision-based tactile sensors.Source: §4.3 · p. 5
4.4. OBJECTFOLDER 1.0 vs. OBJECTFOLDER 2.0
- OBJECTFOLDER 2.0 improves rendering quality across modalities and renders all three in real time; VisionNet is 60× faster than OBJECTFOLDER 1.0.For touch, OBJECTFOLDER 1.0 is compared with DIGIT readings and ours with GelSight readings; TouchNet has the smaller Sim2Real gap.Source: Table 1 · p. 5, Table 2 · p. 6, §4.4 · p. 6
5. Sim2Real Object Transfer
The paper evaluates transfer from rendered data to real objects across scale estimation, contact localization, and shape reconstruction. Experiments use 13 objects overall, including eight real objects for scale estimation.
5.1. Object Scale Estimation
- ObjectFolder 2.0 models generalize better to real objects than ObjectFolder 1.0 models, and touch has the smallest Sim2Real gap.Scale is the longest side of the enclosing axis-aligned bounding box; the scale experiment tests on eight real objects, using ResNet-18 for vision and audio and 10 consecutive touch readings for tactile prediction.Source: §5.1 · p. 6, §5.2 · p. 6, Table 3 · p. 7
5.2. Tactile-Audio Contact Localization
- Touch localizes contact more accurately than audio, while combining both modalities gives the best Sim2Real performance.Particle filtering updates surface-location candidates from tactile FCRN or 3-second impact-sound MFCC features, along with end-effector translations; the pitcher example shows particles converging to ground truth.Source: §5.2 · p. 7, Figure 6 · p. 7, Table 4 · p. 8
5.3. Visuo-Tactile Shape Reconstruction
- Single-image and touch-based reconstructions outperform the average-shape baseline, while combining modalities usually gives the best Sim2Real transfer.The method uses a single image and 32 tactile readings as inputs to point-cloud completion; qualitative results capture the square tray and coffee mug shapes in simulation and real-world experiments.Source: §5.3 · pp. 7–8, Table 5 · p. 8, Figure 7 · p. 8
6. Broader Impact and Limitations
The dataset and code are intended as an accessible standard benchmark, and the implicit representation is cheaper to render. The demonstrated transfer remains bounded to rigid objects with one homogeneous material.
Broader Impact
- The dataset and code are intended to provide an accessible standard benchmark, avoiding real-object purchases for these tasks.The authors note this may especially benefit people facing challenges with international shipping or purchasing specific objects.Source: §6 · p. 8
- Implicit multisensory rendering is computationally cheaper than the initial simulation and may be more environmentally friendly.Source: §6 · p. 8
Limitations
- The demonstrated transfer covers rigid-body objects with a single homogeneous material, not complex objects with non-rigid parts or varied materials.The authors also leave diverse lighting, noise, and reverberation conditions for future work.Source: §6 · p. 8
7. Conclusion
ObjectFolder 2.0 is a 1,000-object dataset of implicit neural representations for multisensory learning in computer vision and robotics. It is 10 times larger and orders of magnitude faster to render than existing work, improves multisensory data quality, and supports transfer to real-world counterparts across three challenging tasks.
- ObjectFolder 2.0 contains 1,000 objects as implicit neural representations for multisensory learning in computer vision and robotics.Source: §7 · p. 8
- The dataset is 10 times larger and orders of magnitude faster to render than existing work, with significantly improved multisensory data quality and realism.Source: §7 · p. 8
- Models learned with its virtualized objects successfully transfer to real-world counterparts on three challenging tasks.Source: §7 · p. 8