Source-linked AI summary
TinyAudio: Compact and Efficient Text-to-Audio Generation for Low-Resource Deployment
Junxi Liu, Xiquan Li, Wenhao Guan, Yifan Duan, Zhikang Niu, Yanru Huo, Ziyang Ma, Xie Chen
TL;DR
Text-to-audio systems often require large models and iterative sampling, limiting deployment on resource-constrained devices. TinyAudio is a compact flow-matching system with audio-aligned conditioning and a one-step MeanFlow variant. It achieves competitive quality with low memory use, and the accelerated variant reaches real-time generation under a four-core CPU quota.
Problem
High-performing text-to-audio systems often rely on large networks and iterative sampling, creating memory and latency demands that limit resource-constrained deployment.
Source: §1 · p. 1Method
TinyAudio combines a single-stream flow-matching Transformer with audio-aligned text conditioning and a compressed-latent waveform decoder, with MeanFlow training for a one-step variant.
Source: §1 · p. 1, §2 · p. 2, §2.3 · p. 2Results
0.48 GB peak GPU memory accompanies competitive AudioCaps and TTA-Bench quality; TinyAudio-MF reaches a steady-state RTF of 0.715 under a four-core CPU quota.
Source: §1 · p. 1, §3.3 · p. 4, §4 · p. 4Takeaways & Limitations
The results demonstrate a practical quality–footprint trade-off for low-resource text-to-audio deployment.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Text-to-audio (TTA) generation has advanced rapidly in generation quality and instruction following. However, representative systems often require around a billion parameters, limiting deployment on resource-constrained devices. This paper introduces TinyAudio, a compact flow-matching-based TTA model for low-resource deployment. At its core, TinyAudio uses TA-DiT, a 35M single-stream flow-matching Transformer. TinyAudio also includes TA-CLAP, a 32M audio-aligned text encoder, and TA-VAE, whose 20M decoder reconstructs 44.1 kHz audio from compressed latents. TinyAudio has only 87M parameters in total, over 90% fewer than representative billion-parameter pipelines, and uses 0.48 GB peak GPU memory. TinyAudio achieves competitive generation quality on AudioCaps and TTA-Bench. We further introduce TinyAudio-MF, a MeanFlow-accelerated model that enables real-time generation with a four-core CPU quota. Our results demonstrate a practical quality-footprint trade-off for low-resource TTA deployment.
1. INTRODUCTION
TinyAudio targets the size and deployment costs of text-to-audio systems with a compact flow-matching pipeline. It combines parameter-efficient text–audio generation and audio-aligned conditioning, with a one-step variant for lower-latency generation.
1. INTRODUCTION
- High-performing text-to-audio systems often use large networks and iterative sampling, increasing memory needs and inference latency for resource-constrained deployment.Efficient sampling alone does not reduce model size; compact systems also need efficient text conditioning, latent generation, and waveform reconstruction.Source: §1 · p. 1
- TinyAudio combines a 35M single-stream TA-DiT, a 32M audio-aligned TA-CLAP encoder, and an approximately 20M TA-VAE decoder in an 87M-parameter pipeline.TA-DiT shares conditional modulation across Transformer blocks; TA-VAE reconstructs 44.1 kHz waveforms from compressed latents.Source: §1 · p. 1
- Quality-aware supervised fine-tuning selects audio–text pairs for perceptual quality and semantic consistency while balancing sampling across audio domains.The paper identifies curated training data as critical to compact TTA model performance.Source: §1 · p. 1
- TinyAudio-MF uses MeanFlow training to reduce latent generation from 25 Euler steps to one step.The one-step variant achieves a steady-state RTF of 0.715 under a four-core CPU quota.Source: §1 · p. 1
- TinyAudio achieves competitive quality on AudioCaps and TTA-Bench with 0.48 GB peak GPU memory.The AudioCaps figure places it in a substantially smaller generator-parameter regime while retaining competitive distributional quality.Source: §1 · p. 1, Figure 1 · p. 1
2. METHOD
TinyAudio combines a compressed audio-latent representation, an audio-aligned text encoder, and a single-stream Transformer generator. The method also uses quality-aware fine-tuning and MeanFlow training to reduce sampling steps.
2. METHOD
- Together, TA-CLAP, TA-DiT, and TA-VAE provide an audio-aware text encoder, a latent generator, and a waveform decoder in an 87M-parameter system.The components contain 32M, 35M, and approximately 20M parameters, respectively.Source: §2 · p. 2
2.1. TA-VAE: Compact Audio Latent Representation
- TA-VAE compresses 44.1 kHz waveforms 1024× into 64-channel latent sequences at 43 Hz.During training its 7M-parameter encoder produces normalized latent targets; at inference, the approximately 20M decoder reconstructs waveforms from generated latents.Source: §2.1 · p. 2
2.2. TA-CLAP: A Lightweight Audio-Aligned Text Encoder
- TA-CLAP adapts a compact text encoder through audio–text contrastive learning, then retains only the text branch for deployment.It pairs the text encoder with a pretrained HTSAT audio encoder and trains normalized embeddings with a pairwise sigmoid loss.Source: §2.2 · p. 2
2.3. TA-DiT: Lightweight Single-Stream Audio Generation Transformer
- TA-DiT concatenates projected audio and text tokens for shared self-attention, using shared attention parameters across modalities rather than separate branches.The joint sequence lets audio and text tokens interact through self-attention.Source: §2.3 · pp. 2–3
- TA-DiT shares conditional modulation networks across Transformer blocks while retaining block-specific embeddings and layer-specific attention and feed-forward parameters.The shared modulation produces adaptive normalization parameters and residual gates for the attention and feed-forward branches.Source: §2.3 · p. 3
2.4. Quality-Aware Supervised Fine-Tuning
- Quality-aware fine-tuning filters audio–text pairs by perceptual and semantic scores, balances music, speech, and general sounds, and retains approximately 417K pairs.Within each category, selection favors higher LAION-CLAP similarity scores.Source: §2.4 · p. 3
2.5. MeanFlow One-Step Acceleration
- TinyAudio-MF continues training from a pretrained TinyAudio checkpoint with the Improved Mean Flows objective, reducing sampling from 25 Euler steps to one function evaluation.It uses the quality-aware SFT subset and standard SFT optimization settings.Source: §2.5 · p. 3
3. EXPERIMENTS
The experiments evaluate TinyAudio on AudioCaps and TTA-Bench, comparing generation quality and deployment cost while testing its architecture, text conditioning, and fine-tuning choices.
3.1. Datasets
- The evaluation uses 10-second AudioCaps test generations and TTA-Bench’s 1,500-prompt Accuracy subset; AudioCaps validation and test audio are excluded from training.The approximately 3.7M-pair corpus supports pretraining, TA-CLAP contrastive learning, and TA-VAE reconstruction; quality-aware SFT uses approximately 417K pairs.Source: §3.1 · p. 3
3.2. Training and Evaluation Details
- Training uses 500K generator-pretraining steps and 200K quality-aware SFT steps, with joint text-condition dropout of 0.1 and 25-step Euler sampling.AudioCaps metrics include FAD, FD, KL, and IS; TTA-Bench reports CE, CU, PC, PQ, and CLAP similarity, while ten experts rate OVL and REL on five-point scales.Source: §3.2 · p. 4
3.3. Generation Quality and Deployment Cost
- 2.65 AudioCaps FAD, close to Tango and MeanAudio; TinyAudio is competitive in TTA-Bench CU and PQ, with higher FD than most compared systems.TinyAudio-MF reduces sampling cost, with a measurable objective-generation-metric quality trade-off.Source: §3.3 · p. 4
- 0.48 GB peak GPU memory is more than 8× less than every compared system; TinyAudio ranks first in mean OVL and second in mean REL, behind TangoFlux.TinyAudio-MF achieves mean steady-state generation RTF 0.715 for 10-second FP32 audio at batch size 1 under a four-core CPU quota; timing includes text encoding and waveform decoding.Source: §3.3 · p. 4
3.4. Ablation Studies
- At matched parameter counts, Base offers a better overall quality trade-off than Per-block AdaLN and Dual-stream variants; removing TA-CLAP adaptation consistently degrades performance.The comparison uses variants trained for 200K steps on AudioSet and AudioCaps, evaluated on 957 AudioCaps test samples.Source: §3.4 · p. 4
- Quality-aware SFT data selection is more effective than random sampling, while additional fine-tuning alone does not guarantee improvement for compact generators.The SFT variants use equally sized 417K-pair subsets with identical initialization and optimization settings.Source: §3.4 · p. 4
- TA-CLAP achieves comparable audio–text retrieval performance to LAION-CLAP in both directions on the AudioCaps test split.The table reports retrieval performance, with LAION-CLAP results re-evaluated by the authors.Source: §3.4 · p. 4, Table 5 · p. 4
4. CONCLUSION
TinyAudio combines a compact flow-matching generator with audio-aligned text conditioning and compressed-latent waveform reconstruction. The conclusion reports competitive quality alongside low-memory and four-core CPU deployment.
4. CONCLUSION
- TinyAudio has 87M inference-time parameters, with TA-DiT combining single-stream text–audio modeling and layer-shared conditional modulation.TA-CLAP provides audio-aligned text conditioning, and TA-VAE reconstructs 44.1 kHz waveforms from compressed latents.Source: §4 · p. 4
- TinyAudio achieves competitive generation quality on AudioCaps and TTA-Bench with 0.48 GB peak GPU memory.Source: §4 · p. 4
- TinyAudio-MF enables real-time generation with a four-core CPU quota, supporting a practical quality–footprint trade-off for low-resource TTA deployment.Source: §4 · p. 4
5. COMPLIANCE WITH ETHICAL STANDARDS
The subjective listening test followed the stated ethical standards.