Source-linked AI summary
GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing
Jian Guo, He He, Tong He, Leonard Lausen, Mu Li, Haibin Lin, Xingjian Shi, Chenguang Wang, Junyuan Xie, Sheng Zha, Aston Zhang, Hang Zhang, Zhi Zhang, Zhongyue Zhang, Shuai Zheng, Yi Zhu
TL;DR
GluonCV and GluonNLP address the need for practical, reusable deep-learning tooling in computer vision and natural language processing. They combine modular APIs, pre-trained models, training resources, and MXNet deployment support, and their implementations match or outperform comparable open-source implementations across evaluated benchmarks. The toolkits therefore support rapid experimentation and downstream application development.
Problem
Deep-learning development needs tooling that supports customization, rapid prototyping, reproducibility, and deployment across computer vision and natural language processing tasks.
Method
The paper develops MXNet-based GluonCV and GluonNLP toolkits with modular APIs, model-zoo resources, and cross-language deployment support.
Results
GluonCV/GluonNLP matches or outperforms compared open-source implementations for the same models on the same datasets across evaluated tasks.
Takeaways & Limitations
The toolkits allow users to rapidly try new ideas and develop downstream applications in computer vision and natural language processing.
Abstract
from arXiv · showhide
We present GluonCV and GluonNLP, the deep learning toolkits for computer vision and natural language processing based on Apache MXNet (incubating). These toolkits provide state-of-the-art pre-trained models, training scripts, and training logs, to facilitate rapid prototyping and promote reproducible research. We also provide modular APIs with flexible building blocks to enable efficient customization. Leveraging the MXNet ecosystem, the deep learning models in GluonCV and GluonNLP can be deployed onto a variety of platforms with different programming languages. The Apache 2.0 license has been adopted by GluonCV and GluonNLP to allow for software distribution, modification, and usage.
1. Introduction
GluonCV and GluonNLP are MXNet-based toolkits designed to make deep-learning development in computer vision and natural language processing more customizable, reproducible, and deployable.
- Deep learning frameworks have enabled rapid advances in computer vision and natural language processing.
- GluonCV/NLP provide modular APIs that support customization by reusing efficient building blocks.
- Pre-trained state-of-the-art models, training scripts, and training logs support fast prototyping and reproducible research.
- The toolkits’ models can be deployed through MXNet in C++, Clojure, Java, Julia, Perl, Python, R, and Scala.
- Separate GluonCV and GluonNLP frontends let users install either toolkit independently while sharing the MXNet backend and its language bindings.
2. Design and Features
The toolkits combine reusable APIs, broad data access, a model zoo, and MXNet-based deployment support to streamline development across vision and language tasks.
- 2.1 Modular APIs: Modular APIs let users customize model design, training, and inference by reusing data utilities, model components, initialization methods, and loss functions.
- 2.1 Modular APIs: Data utilities transform variable-shaped inputs and labels into efficiently computed mini-batches.
- 2.1 Modular APIs: Fixed bucketing groups similarly shaped inputs together, reducing padding and saving computation.
- 2.1 Modular APIs: The data API provides access to datasets including ImageNet, VOC, COCO, SST, IWSLT, SQuAD, and WikiText.
- 2.2 Model Zoo: The model zoo offers pre-trained models, training scripts, and logs for more than 200 computer vision and natural language processing models and tasks.
- 2.3 Leveraging the MXNet Ecosystem: MXNet supports CPU, GPU, and multi-machine training, while hybridization enables deployment across multiple programming languages with minimal configuration.
3. Performance
Across standard computer vision and natural language processing benchmarks, GluonCV/NLP implementations generally match or outperform other open-source implementations of the same models.
- The evaluation compares GluonCV/NLP with Caffe, Caffe2, Theano, and TensorFlow implementations across popular vision and language tasks.
- GluonCV/GluonNLP matches or outperforms compared open-source implementations for the same model evaluated on the same dataset.
- ResNet-50 image classification on ImageNet is one case where the GluonCV implementation significantly outperforms implementations in other frameworks.
4. Conclusion
GluonCV/NLP combine modular APIs and a model zoo to support rapid experimentation and downstream application development, while continuing to expand their capabilities.
- Modular APIs and the model zoo allow users to rapidly try new ideas or develop downstream applications in computer vision and natural language processing.
- Planned future work includes enriching the APIs and model zoo, accelerating inference, improving NumPy compatibility, and supporting deployment in more scenarios.