Source-linked AI summary
TorchAudio: Building Blocks for Audio and Speech Processing
Yao-Yuan Yang, Moto Hira, Zhaoheng Ni, Anjali Chourdia, Artyom Astafurov, Caroline Chen, Ching-Feng Yeh, Christian Puhrsch, David Pollack, Dmitriy Genzel, Donny Greenberg, Edward Z. Yang, Jason Lian, Jay Mahadeokar, Jeff Hwang, Ji Chen, Peter Goldsborough, Prabhat Roy, Sean Narenthiran, Shinji Watanabe, Soumith Chintala, Vincent Quenneville-Bélair, Yangyang Shi
TL;DR
Audio and speech machine-learning development needs domain-specific building blocks that are efficient, differentiable, and deployable. TorchAudio provides such components across I/O, datasets, operations, and canonical models, and benchmarks report parity or better performance against public implementations.
Problem
Developing audio and speech machine-learning applications requires domain-specific functionality beyond general-purpose open-source toolkits.
Method
TorchAudio provides PyTorch-compatible building blocks designed for GPU computation, automatic differentiation, production readiness, and multiple audio/speech functionalities.
Results
Benchmarks of five audio/speech operations and three models show parity or better runtime performance and output quality than publicly available reference implementations.
Takeaways & Limitations
TorchAudio supports development and deployment of audio-related machine-learning applications through broad functionality and active community use.
Abstract
from arXiv · showhide
This document describes version 0.10 of TorchAudio: building blocks for machine learning applications in the audio and speech processing domain. The objective of TorchAudio is to accelerate the development and deployment of machine learning applications for researchers and engineers by providing off-the-shelf building blocks. The building blocks are designed to be GPU-compatible, automatically differentiable, and production-ready. TorchAudio can be easily installed from Python Package Index repository and the source code is publicly available under a BSD-2-Clause License (as of September 2021) at https://github.com/pytorch/audio. In this document, we provide an overview of the design principles, functionalities, and benchmarks of TorchAudio. We also benchmark our implementation of several audio and speech operations and models. We verify through the benchmarks that our implementations of various operations and models are valid and perform similarly to other publicly available implementations.
1. INTRODUCTION
TorchAudio provides domain-specific building blocks for audio and speech machine learning, emphasizing GPU capability, automatic differentiation, production readiness, stability, and broad functionality. Benchmarks report parity or better performance against public implementations, alongside substantial community adoption.
- TorchAudio was established to accelerate development of machine learning applications in the audio and speech domain.
- Each building block targets GPU compute capability, automatic differentiability, and production readiness.These properties support accelerated training and inference, end-to-end learning, and deployment across environments including mobile devices.
- The toolkit prioritizes stability and provides audio I/O, datasets, audio/speech operations, and canonical machine learning models.
- Benchmarks of five operations and three models report parity or better performance against publicly available reference implementations.Operations achieve or exceed runtime parity, while models achieve or exceed output-quality parity.
- By September 2021, the project had addressed more than 400 issues, merged more than 1250 pull requests, and had more than 5,420 dependent public GitHub repositories.
2. RELATED WORK
TorchAudio occupies the PyTorch ecosystem alongside audio and speech toolkits built for NumPy and TensorFlow. It supplies lower-level functionality and interoperable access to datasets within PyTorch-based workflows.
- Users commonly commit to one ecosystem because cross-ecosystem dependencies and conversions increase maintenance cost.
- TorchAudio is designed for PyTorch, while librosa serves NumPy and DDSP and TensorFlowASR serve TensorFlow.
- TorchAudio provides audio I/O, spectrogram computation, and a unified dataset interface as low-level PyTorch functionality.
- Other PyTorch ecosystem toolkits provide ready-to-use models for applications including recognition, enhancement, separation, speaker recognition, and text-to-speech.
3. DESIGN PRINCIPLES
TorchAudio’s design principles combine PyTorch integration with GPU compatibility, automatic differentiation, and production deployment support. The project also emphasizes stability, selective scope, testing, and platform coverage.
- TorchAudio is built to integrate with PyTorch neural-network containers and data-loading utilities without dependencies beyond PyTorch.
- Most functionalities are designed for GPU computation, automatic differentiation, and TorchScript compilation for production use.TorchScript supports serialization and execution in environments without Python, including C++ systems and mobile platforms.
- The toolkit supports PyTorch’s major platforms and Python versions 3.6 through 3.9, with stable features intended to preserve backward compatibility when possible.
- Release candidates collect user feedback before official releases, while backward-compatible-breaking changes to stable features receive restricted release treatment.
- TorchAudio uses documented code, continuous integration, and more than 6000 pytest test cases to support codebase quality.
- Feature selection favors essential functionality and canonical baseline models such as Tacotron2 to keep the package lean and stable.
4. PACKAGE STRUCTURE AND FUNCTIONALITIES
TorchAudio spans audio I/O, datasets, operations, and machine learning models within a PyTorch-compatible package structure. Its components cover common processing tasks and canonical downstream applications.
- TorchAudio supports four functionality categories: audio I/O, audio/speech datasets, audio/speech operations, and audio/speech models.
- The backend submodule loads audio files into PyTorch tensors and saves tensors to audio files, with TorchScriptable SoX support and optional alternative backends.
- The dataset submodule provides access to 11 commonly used datasets through interfaces compatible with PyTorch DataLoader.
- Audio operations are organized into functional, transform, and sox_effects submodules.
- The functional submodule includes 49 operations spanning utilities, filtering, and feature extraction, while transform provides 26 neural-network-compatible transforms.
- The sox_effects module ports SoX functionality into TorchAudio, is fully TorchScriptable, and supports 58 sound effects.
- Canonical models cover speech recognition, speech separation, text-to-speech, and neural vocoding, including Wav2Vec 2.0, Conv-TasNet, Tacotron2, and WaveRNN.
5. EMPIRICAL EVALUATIONS
The evaluation benchmarks five audio/speech operations and three canonical machine-learning models against publicly available implementations. TorchAudio generally matches or exceeds reference runtime and output quality, with GPU acceleration and marginal JIT trade-offs.
- Evaluation setup: The study benchmarks five audio/speech operations by runtime and three machine-learning models using task-specific quality metrics.Experiments were conducted on an AWS p4d.24xlarge instance with NVIDIA A100 GPUs.
- 5.1. Audio/Speech Operations: TorchAudio matches or slightly exceeds librosa on CPU runtime across the tested operations and additionally supports GPU acceleration.The compared operations include phase vocoder, Griffin-Lim, MFCC, spectral centroid, and spectrogram.
- 5.1. Audio/Speech Operations: JIT compilation is slightly slower on CPU and slightly faster on GPU, but the difference is marginal.The comparison uses TorchAudio’s regular, JIT-compiled, and GPU-accelerated implementations.
- 5.2. Audio/Speech Applications: TorchAudio’s WaveRNN achieves similar PESQ, STOI, and MOS performance to fatchord’s WaveRNN and Nvidia’s WaveGlow.MOS is evaluated using human ratings, while PESQ and STOI assess waveform reconstruction quality.
- 5.2. Audio/Speech Applications: The Tacotron2 implementations differ by around 1% in MCD, supporting the validity of TorchAudio’s implementation.MCD is evaluated on the LJSpeech dataset against Nvidia’s implementation.
- 5.2. Audio/Speech Applications: TorchAudio’s Conv-TasNet slightly outperforms Asteroid on both Si-SDRi and SDRi metrics.The model is trained on the sep_clean task of Libri2Mix at an 8000 Hz sample rate.
6. CONCLUSION
TorchAudio is presented as a toolkit under active development for accelerating audio-related machine-learning development and deployment. The project’s roadmap is maintained on GitHub.
- 6. CONCLUSION: TorchAudio is under active development to accelerate development and deployment of audio-related machine-learning applications.The paper describes the toolkit as a brief summary of TorchAudio’s functionality and direction.