Source-linked AI summary
Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research
Krishna Murthy Jatavallabhula, Edward Smith, Jean-Francois Lafleche, Clement Fuji Tsang, Artem Rozantsev, Wenzheng Chen, Tommy Xiang, Rev Lebaredian, Sanja Fidler
TL;DR
3D deep learning research lacks standardized tools spanning data, representations, models, rendering, evaluation, and visualization. Kaolin addresses this gap with a PyTorch library integrating these capabilities, while its main scope boundary is incomplete support for secondary lighting effects and 3D object detection.
Problem
3D deep learning research lacks standardized tools for efficiently handling representations, geometry, rendering, evaluation, and visualization.
Method
Kaolin integrates PyTorch-based data handling, 3D representation operations, geometric functions, modular differentiable rendering, metrics, model architectures, and visualization utilities.
Results
Kaolin provides a comprehensive toolkit covering the main lifecycle of 3D deep learning applications, from data preparation through modeling, evaluation, and visualization.
Takeaways & Limitations
The package aims to reduce boilerplate and make 3D deep learning research and application development more intuitive and approachable.
Takeaways & Limitations
Kaolin currently supports only primary lighting effects and lacks model-zoo support for 3D object detection.
Abstract
from arXiv · showhide
We present Kaolin, a PyTorch library aiming to accelerate 3D deep learning research. Kaolin provides efficient implementations of differentiable 3D modules for use in deep learning systems. With functionality to load and preprocess several popular 3D datasets, and native functions to manipulate meshes, pointclouds, signed distance functions, and voxel grids, Kaolin mitigates the need to write wasteful boilerplate code. Kaolin packages together several differentiable graphics modules including rendering, lighting, shading, and view warping. Kaolin also supports an array of loss functions and evaluation metrics for seamless evaluation and provides visualization functionality to render the 3D results. Importantly, we curate a comprehensive model zoo comprising many state-of-the-art 3D deep learning architectures, to serve as a starting point for future research endeavours. Kaolin is available as open-source software at https://github.com/NVIDIAGameWorks/kaolin/.
1. Introduction
Kaolin addresses the steep learning curve caused by missing standardized tools in 3D deep learning. It provides a PyTorch library covering the workflow from data preparation through modeling, rendering, evaluation, and visualization.
- Kaolin targets the lack of standardized tools that makes 3D deep learning research difficult to conduct.
- The library provides efficient PyTorch implementations for loading data, converting 3D representations, performing tasks, computing metrics, and visualizing results.
- Kaolin includes popular 3D methods and pre-trained models in a model zoo to support implementation and future research.
- A modular differentiable renderer supports geometric transformations, lighting, shading, projection, and extensions to rendering methods.
2. Kaolin - Overview
Kaolin combines broad 3D data, geometry, rendering, evaluation, model-zoo, and visualization capabilities in a PyTorch-oriented research toolkit. Its modular operations and dataset interfaces aim to make 3D deep learning workflows more accessible and reusable.
- 3D Representations: Kaolin supports meshes, pointclouds, voxel grids, signed distance functions, level sets, and depth images as PyTorch Tensor-based representations.It provides efficient, wherever possible differentiable, conversions between representations, including mesh-to-pointcloud surface sampling.
- Datasets: Kaolin simplifies loading popular 3D datasets through PyTorch Dataset and DataLoader extensions and supports USD import and export.Supported collections include ShapeNet, PartNet, SHREC, ModelNet, ScanNet, and HumanSeg; datasets without open licenses require independent download.
- 3D Geometry Functions: Kaolin provides geometric transformations, image warping, projection, and backprojection for manipulating 3D content.Supported transformations include rigid-body, affine, and projective operations, with orthographic and perspective projection.
- Modular Differentiable Renderer: Its differentiable renderer separates geometric transformations, lighting, shading, rasterization, and projection into swappable modules.Kaolin supports multiple lighting, shading, projection, and rasterization modes, with CUDA implementations where necessary.
- Loss Functions and Metrics: Kaolin includes representation-specific losses and metrics, including voxel intersection over union, pointcloud Chamfer distance, and mesh regularizers.
- Model-zoo and Visualization: The model zoo offers baselines, state-of-the-art architectures, source code, and pre-trained models across classification, segmentation, reconstruction, super-resolution, and rendering.The package also provides visualization across all supported representations, including headless rendering of images and animations to disk.
3. Roadmap
Kaolin’s roadmap prioritizes expanding its model zoo, differentiable rendering, dataset coverage, and mixed-precision support while addressing current gaps in 3D object detection and secondary lighting.
- Model Zoo: The model zoo is slated for continual improvement because Kaolin reduces implementation time for new methods.The roadmap estimates that most approaches can be implemented in a day or two.
- Differentiable rendering: Future releases plan broader differentiable-rendering support, including domain randomization and material recovery.
- LiDAR datasets: Kaolin plans to add large-scale LiDAR semantic and instance-segmentation datasets, prioritizing S3DIS and nuScenes.
- 3D object detection: 3D object-detection models are currently absent from Kaolin’s model zoo and are identified as a future-release focus.
- Automatic Mixed Precision: The roadmap investigates Automatic Mixed Precision for common 3D architectures to make them more compact and fast.
- Secondary light effects: Current differentiable rendering supports only primary lighting effects, limiting reasoning about complex scene information such as shadows.Future releases plan path-tracing and ray-tracing support for secondary effects.