PaperlayerView exact revision on arXiv

Source-linked AI summary

A Quantized Native Runtime for On-Device Semantic Audio Generation

Matteo Spanio, Antonio Rodà

arXiv:2607.08526v1cs.SDcs.PF

TL;DR

Semantic audio deployment needs controllable generation beyond Python/PyTorch and discrete-GPU stacks. aria addresses this with a dependency-free native runtime and quantization study, finding that 8-bit precision preserves measured quality while 4-bit enables Stable Audio 3 on an 8 GB Raspberry Pi.

  • Problem

    Semantic audio systems need portable, predictable, low-latency generation on commodity and embedded hardware, but existing implementations assume Python/PyTorch and discrete GPUs.

  • Method

    The paper builds a dependency-free native runtime for Stable Audio 3 and evaluates memory-frugal quantization and activation steering with independent quality checks.

  • Results

    8-bit precision keeps prompt adherence, distributional quality, and taste within re-seed noise, while 4-bit runs the 1.2-billion-parameter model on an 8 GB Pi.

  • Takeaways & Limitations

    aria provides a practical compact runtime for on-device semantic audio, while steering offers genuine control only within a narrow window for a subset of attributes.

  • Takeaways & Limitations

    The steering study uses automatic oracles for one model family, so perceptual confirmation through human listening remains open.

Abstract

from arXiv · show

Semantic audio applications increasingly require controllable generation on commodity and embedded hardware rather than through framework-heavy datacenter stacks. We present \textit{aria}, a dependency-free native runtime that runs the complete text-to-music pipeline of Stable Audio~3 (SA3) on ordinary GPUs, CPU-only machines, and a Raspberry~Pi~5, with no Python or deep-learning framework underneath. Our main contribution is a study of quantization: running the model at lower numerical precision to fit tight memory budgets, saving memory in place rather than adding to it. Because the runtime owns every internal tensor, it also exposes activation steering, a low-cost way to steer what the model generates. We judge the quality cost with three independent measures of the output (prompt adherence, overall audio quality, taste preservation), each compared against the ordinary variation between random seeds. Eight-bit precision shows no measurable quality loss on any measure while sharply cutting memory, and it is the fastest mode on the GPU; four-bit adds a small, bounded cost but shrinks the footprint enough to run the $1.2$-billion-parameter model on an $8$\,GB Pi. Against the official implementation, aria matches or exceeds generation speed and starts about seven times faster. A case study of the steering interface generates music carrying taste associations (\emph{sonic seasoning}), with genuine but bounded control for a subset of attributes. These results make a compact, quantized runtime with built-in control a practical basis for on-device semantic audio in Internet-of-Sounds settings. The \textit{aria} runtime is released at https://github.com/matteospanio/aria.

I. INTRODUCTION

The introduction frames on-device semantic audio as a deployment challenge and presents aria, a dependency-free native SA3 runtime designed for portable, memory-bounded generation and built-in activation steering. It uses sonic seasoning to test whether runtime steering can produce genuine semantic control.

  • Motivation: On-device semantic audio requires quality alongside cold-start latency, predictable memory, portability across CPUs and GPUs, and resident models near users.The motivation spans interactive tools, local creative services, and embedded audio devices.
  • Motivation: Open music models of roughly 1–10 B parameters and diffusion-based inference make audio generation a strong candidate for moving from cloud systems to owned hardware.Stable Audio 3 is selected as the reference target among leading open-source diffusion-transformer music generators.
  • Runtime: aria is a dependency-free C/CUDA runtime executing SA3’s complete text-to-music pipeline on commodity GPUs, CPU-only systems, and embedded-class hardware.The native stack includes the tokenizer, text encoder, transformer denoiser, and autoencoder, and supports both model variants.
  • Quantization: 0.84 GB is the 8-bit Raspberry Pi 5 peak memory for a 10 s stereo clip, down from 1.9 GB at full precision for the small model.The runtime stores weights from half precision to 8- or 4-bit integers, uses 8-bit arithmetic where beneficial, and targets bounded long-form memory use.
  • Activation steering: Activation steering becomes a runtime primitive because aria owns every intermediate tensor, enabling negligible-cost semantic directions injected into the generation graph.Sonic seasoning, or taste-conditioned music generation, serves as a stress test because taste is difficult to specify lexically and can distinguish semantic change from degradation.

II. RELATED WORK · A. Generative audio models · B. Generative models on edge devices

Related work spans latent-diffusion text-to-music systems and efforts to bring generative models onto edge devices. The paper positions aria within open-weight, self-contained runtimes that support low-bit quantization and resident execution.

  • A. Generative audio models: Text-to-music systems comprise autoregressive transformers over discrete acoustic tokens and latent-diffusion models operating in neural-audio-autoencoder latent spaces.Diffusion cost scales with denoising steps rather than sequence length.
  • A. Generative audio models: Leading open-weight systems couple continuous audio latents to diffusion-transformer denoisers conditioned on text.The described design uses a SAME semantic-acoustic autoencoder, a 256-dimensional latent, DiT denoising, and T5Gemma conditioning.
  • A. Generative audio models: The study covers small-music and medium configurations with 20 and 24 DiT blocks, respectively.Their model dimensions are dmodel=1024 and dmodel=1536, respectively.
  • A. Generative audio models: The DiT residual stream provides the hidden-state substrate for activation steering.Activation steering reads and writes per-block hidden states.
  • A. Generative audio models: The shift toward live generation, exemplified by Magenta / Lyria RealTime, increases the appeal of warm, resident runtimes over cold per-request processes.This comparison motivates runtime designs that remain loaded between generations.
  • B. Generative models on edge devices: State-of-the-art generative audio increasingly relies on closed APIs such as Suno and Udio, while open-weight reference stacks are typically heavyweight Python pipelines assuming discrete GPUs.Community CPU ports of SA3 remain tied to PyTorch.
  • B. Generative models on edge devices: The ggml lineage demonstrates a contrasting single, self-contained C/C++ runtime approach for generative models on edge devices.Examples include llama.cpp, whisper.cpp, DwarfStar, and stable-diffusion.cpp, alongside memory-mapped weights and low-bit quantization.

C. Quantization for on-device diffusion · D. Steering techniques · III. METHODOLOGY

The methodology positions low-bit inference as the key to fitting native diffusion runtimes onto constrained hardware, while activation interventions provide training-free control of frozen generative models. It builds on prior quantization and steering methods, including audio-specific PTQ and injection designs for music generation.

  • C. Quantization for on-device diffusion: Low-bit inference is presented as the lever for fitting diffusion runtimes within hardware memory budgets.The passage frames quantization as a deployment strategy rather than solely a fixed-fidelity optimization.
  • C. Quantization for on-device diffusion: Prior PTQ work compresses language models to 8-bit weights and activations, diffusion image models to 4-bit weights, and audio diffusion transformers to lower precision.These examples establish precedents across language, image, and audio generation.
  • C. Quantization for on-device diffusion: Unlike work targeting a fixed fidelity target, this methodology treats numerical precision as a first-class deployment axis.The supplied passage introduces this contrast but truncates the runtime’s full precision range.
  • D. Steering techniques: Activation-based steering controls a frozen generative model by adding an interpretable attribute direction to its residual stream during inference.This approach follows the linear representation hypothesis, which treats attributes as approximately linear directions in hidden space.
  • D. Steering techniques: Activation addition and representation engineering extract steering directions from contrastive examples and inject them at generation time.The passage notes that Tan et al. characterize when such vectors generalize and that the paradigm is established in audio and music.
  • D. Steering techniques: The audio steering design follows Camporese et al., while related methods include LoRA, ControlNet, and concept sliders.The weight-based family can provide strong control but requires training and per-attribute data.

A. The aria runtime · B. Taste steering and sonic seasoning

aria is a dependency-free native Stable Audio 3 runtime designed for interactive, budget, and CPU-only inference, with quantization and streaming that reduce deployment costs. Its built-in activation steering supports taste-associated music generation, evaluated with a protocol intended to distinguish genuine control from metric gaming.

  • A. The aria runtime: 7.7k lines of C and CUDA implement aria as a single-purpose, dependency-free Stable Audio 3 engine for interactive, budget, and CPU-only inference.It targets settings where framework startup, GPU-context setup, and multisecond cold starts dominate short generations.
  • A. The aria runtime: The runtime provides vectorized multithreaded CPU and GPU backends, with the CPU path serving as both correctness reference and CPU-only production target.The CPU backend includes an 8-bit-integer matrix-multiply variant, while the GPU backend uses half-precision tensor cores.
  • A. The aria runtime: 3.1× faster per chunk on the Pi is achieved by streaming only emitted frames, while preserving byte-identical output.The streaming variant supports long-form audio generation and streaming on Raspberry Pi 5.
  • A. The aria runtime: 0.60 vs. 1.23 s per job is achieved by resident batch mode and an HTTP server, which keep the model loaded and roughly double throughput at identical output.These modes support multi-client serving on Raspberry Pi 5.
  • A. The aria runtime: Activation steering is built into aria because it owns every intermediate tensor, applying precomputed direction vectors at three pipeline locations.Directions can be added or projected, optionally during selected sampling spans, at the transformer residual stream, audio latent, or text conditioning.
  • A. The aria runtime: q8 and q4 compress transformer weights, while W8A8 also quantizes activations and arithmetic; freeing full-precision sources makes lower precision reduce resident memory.The modes target GPU integer tensor cores or equivalent ARM instructions.
  • A. The aria runtime: Warm generation, invocation, and cold start separately measure resident-service cost, one-shot setup cost, and first-use edge-endpoint cost.The protocol distinguishes per-request inference from process setup, weight transfer, launch, loading, and generation.
  • B. Taste steering and sonic seasoning: Sonic seasoning tests whether steering a frozen music generator with difference-in-means directions can produce associations across sweet, sour, bitter, salty, and spicy tastes.Because the taste oracle is susceptible to metric gaming, the evaluation uses a multi-oracle protocol to distinguish genuine control from degradation that artificially raises the target.

1) Direction and injection site:

The method steers generation using unit directions estimated from contrastive prompt or audio sets and injected into DiT block updates, whose outputs remain in the input model space. Audio-side contrasts use 377 rated music clips to capture taste-related residuals from heard examples.

  • Direction and injection site:: DiT blocks inject steering additively because each block output lies in the same dmodel space as its input.This makes the injection site the block update itself rather than a separate representation space.
  • Direction and injection site:: Unit directions d^(ℓ)_i point from “low-i” to “high-i” and are estimated by difference-in-means over each contrastive set S±.Difference-in-means is described as the simplest estimator known to transfer.
  • Direction and injection site:: Audio-side contrasts rank 377 music clips by per-clip basic-taste ratings and feed the top-/bottom-k clips to SA3 as initial audio inputs.The resulting residuals are grounded in audio that humans actually heard as sweet or sour.

2) Additive injection: · 3) Training-based baseline: per-axis LoRA: · 4) Multi-oracle evaluation:

The method steers one transformer block by adding a scaled direction, contrasts this training-free intervention with per-axis LoRA, and evaluates effects using multiple independent quality and degradation oracles. Genuine steering requires aligned target and CLAP gains at low drift with bounded FAD, while high-strength gains that coincide with quality collapse are degradation.

  • 2) Additive injection:: At inference, steering adds a scaled copy of a direction to one chosen block’s output using a single strength α ≥0.The intervention is a rank-one addition with one hyperparameter.
  • 2) Additive injection:: α = 0 preserves the pass and provides a matched, seed-identical baseline; useful effects occur only at low strength, while larger α severely degrades audio.The sweep reaches α = 1.0 because responses are non-monotonic.
  • 3) Training-based baseline: per-axis LoRA:: Per-axis LoRA (r = 8) trains on the same contrasts with the backbone frozen, providing a training-based baseline for quantifying training-free steering’s sacrifices.It uses a trigger-token objective distinguishing high-i clips captioned “i taste” from neutral “music”.
  • 3) Training-based baseline: per-axis LoRA:: Unlike steering, LoRA requires a per-axis optimization run, adds served parameters, and cannot be removed by zeroing a scalar.Both methods share extraction data, pipeline, and evaluation, with each method’s cost logged.
  • 4) Multi-oracle evaluation:: The target effect Δ_i(α) compares wav2taste scores against the matched α = 0 baseline, but the learned regressor can over-score out-of-distribution audio.Held-out macro Pearson r ≈0.67, with best sweet 0.82 and worst sour 0.59.
  • 4) Multi-oracle evaluation:: Three independent checks—CLAP, FAD, and audio drift—guard wav2taste after loudness normalization to −14 LUFS, preventing loudness from moving any metric.FAD detects degradation over CLAP embeddings, while drift measures perturbation magnitude relative to the α = 0 output.
  • 4) Multi-oracle evaluation:: A setting is genuine only when Δ_i and CLAP rise together at low drift and bounded FAD; divergent gains with worsening FAD or drift are reported as degradation.Figure 2 illustrates wav2taste remaining high while CLAP collapses and FAD explodes beyond the genuine low-α window.
  • 4) Multi-oracle evaluation:: Layer sweeps show apparent taste peaks can be misleading: SWEET peaks at L16, intense tastes spike at the final layer, but quality control identifies L14–17 as genuine.Medium peaks at L20–21.

IV. EXPERIMENTS AND RESULTS · A. Experimental setup

The experiments evaluate steering, runtime reproduction, efficiency, and quantization under controlled prompts, seeds, layers, precision settings, and independent audio-quality checks. The setup separates genuine steering effects from metric gaming and precision-induced variation.

  • A. Experimental setup: All experiments use 10-second clips, −14 LUFS loudness normalization, three seeds, and means, with steering strength α scaled by the patched block’s mean residual norm.Directions come from norm-sonic-seasoning, and experiments use small-music in fp32 and medium in fp16 because medium fp32 exceeds the 8 GB reference GPU.
  • A. Experimental setup: A four-oracle dense-window analysis distinguishes genuine steering from metric gaming: target and CLAP rise together only at low α, while later CLAP reversal and FAD increases accompany continued target growth.Salty barely moves CLAP even when clean, and spicy is marginal; the consolidated ablation table reports ΔW2T, ΔCLAP, and FAD against matched α = 0 baselines.
  • A. Experimental setup: Steering directions are compared by dose-response monotonicity, over-steer robustness, and logistic-probe layer separability before every DiT block is tested at α = 0.15.Blocks are ranked using the target metric alone, after which one candidate block per axis is submitted to the oracle panel.
  • A. Experimental setup: The dense-window study sweeps α ∈ {0.1, 0.15, 0.2, 0.3, 0.5, 0.7, 1.0} across 12 prompts and three seeds using wav2taste, CLAP, FAD, and drift.The medium model repeats the sweet sweep at its own re-scanned best layer.
  • A. Experimental setup: Three ablation families test injection operation, denoising-step window, and steering site using the dense-window panel on small-music.They compare additive, projection, amplifying or removing the existing direction component; early versus late steps; and residual, latent, or pooled-text locations.
  • A. Experimental setup: aria reproduces the dense-window steering experiment with identical direction vectors and α values, comparing Pearson r and MAE against the SA3/PyTorch reference.Efficiency is measured on RTX 3070 and CPU-only x86 systems under warm, invocation, and cold protocols, using the official Stable Audio 3 implementation as baseline.
  • A. Experimental setup: Quantization fidelity is measured against fp16 while keeping the decoder at half precision, isolating transformer weight q8, q4, or weight-plus-activation W8A8 effects.Each precision uses 24 genre-diverse prompts, three seeds, 72 clips, 10-second outputs, and loudness normalization.
  • A. Experimental setup: Quality is assessed with prompt adherence, overall audio quality, and taste preservation using independent CLAP and wav2taste models.Taste preservation is the distance between each clip’s 5-D taste vector and its fp16 counterpart.

B. Runtime benchmarks

aria matches or exceeds the official implementation in warm generation speed, starts 7.2–7.7× faster, and uses 1.4–1.7× less peak GPU memory. Eight-bit quantization is nearly free, while live steering tracks scheduled taste control without extra Raspberry Pi 5 runtime cost.

  • Deployment speed and memory: 7.2–7.7× faster cold starts and 1.4–1.7× smaller peak GPU-memory footprint distinguish aria from the official implementation.In warm operation, a steered 10 s clip runs in 0.13 vs. 0.146 s on small-music and 0.37 vs. 0.443 s on medium.
  • Deployment speed and memory: 1.12 s with W8A8 is aria’s medium-tier 60 s GPU time, improving on 1.28 s at equal precision and the reference’s 1.38 s.On small-music, the official path remains ahead at 0.52 vs. 0.38 s, or 1.37×.
  • Deployment speed and memory: 1.68× faster CPU performance corresponds to 48.0 vs. 80.65 s, while the small-tier CPU gap to the PyTorch baseline essentially closes at 1.09×.The official warm benchmark incurs 14.5–48.1 s of recompilation penalties for new clip lengths, and its medium loader briefly needs approximately 7.1 GB.
  • Quantization: 8-bit quantization is close to free in quality cost and enables edge deployment, evaluated using prompt adherence, FAD, and wav2taste L2.The metrics are designed to avoid relying solely on the taste oracle optimized by the steering study.
  • Live steering: ρ=0.78 with p=0.003 shows that scheduled SWEET steering tracks per-chunk taste across a 12-chunk stream, while a steered Raspberry Pi 5 clip costs 32.9 vs. 32.8 s unsteered.The taste trajectory lags during descent because the continuation inherits context, and semantic control remains fully on-device.

C. Steering results

Steering is effective only within attribute- and site-specific clean operating windows, which depend on direction, layer, model scale, and injection operator. The runtime reproduces these dose–response conclusions while projection amplification and compact-latent injection improve selected quality–control tradeoffs over alternatives.

  • Direction selection: Audio-derived directions are preferred: they reach the SWEET peak while remaining monotonic, unlike the prompt-derived direction’s inverted-U response.The audio-derived direction has Spearman ρ=+0.94 and remains positive at α=0.2.
  • Clean operating window: At α=0.1, SOUR steering produces a genuine shift (∆w2t=+0.42, p=2.9×10−11, d=3.71), but stronger doses make wav2taste and CLAP diverge.At α=1.0, the audio collapses into noise-like output.
  • Layer selection: Layer selection is attribute-dependent: SWEET peaks at mid-late L16, whereas intense tastes rank the final block first despite quality degradation there.Linear separability from logistic probes does not imply steerability.
  • Model scale: The medium backbone widens the clean window, reaching SWEET’s peak at ∆w2t=+0.143 with p=8.9×10−9 and d=1.32 while retaining positive CLAP farther out.At α=0.1, medium degrades less on intense tastes, with lower FAD across axes.
  • Runtime reproduction: aria reproduces both genuine and degradation regimes, with pooled wav2taste r=0.95 for small and 0.92 for medium; CLAP agrees at r=0.95 on medium and r=0.86 on small.The small-model CLAP difference reflects aria’s half precision versus the full-precision reference and vanishes on medium.
  • Ablations and steering sites: Projection amplification matches additive steering’s SOUR peak at lower FAD, while compact-latent injection steers SOUR cleanly but fails SWEET; text-embedding injection fails both axes.SWEET responds only to mid-late residual injection, whereas SOUR is steerable at latent and residual sites.

V. DISCUSSION · A. Implications of warm-parity edge deployment · B. Learned oracles require independent checks

The discussion argues that warm, resident deployment makes aria especially useful for interactive and embedded settings, while learned-metric interventions require independent quality checks to avoid mistaking degradation for success.

  • A. Implications of warm-parity edge deployment: On warm throughput, aria is within a few percent of the official stack and marginally ahead.Its advantage comes less from faster arithmetic than from removing framework startup.
  • A. Implications of warm-parity edge deployment: Framework startup, GPU-context setup, and per-length compilation dominate short generations.Removing these costs explains aria’s warm-throughput advantage.
  • A. Implications of warm-parity edge deployment: Interactive and embedded settings benefit most from a resident, warm model.Cold start, memory footprint, and predictable behavior across clip lengths matter more there than steady-state speed.
  • A. Implications of warm-parity edge deployment: On medium GPUs at exact precision, aria overtakes the official path, while CPU-only rendering is faster for the same clip.Long-form generation on the small GPU tier remains aria’s one stated deficit.
  • B. Learned oracles require independent checks: An intervention that maximizes a learned metric can make that metric unreliable when intervention strength becomes too aggressive.The target may keep rising while causally independent checks collapse.
  • B. Learned oracles require independent checks: Layer selection poses the same risk because the target can mistake degradation for success.Both intervention strength and intervention site require supervision by quality signals sharing no training path with the target.

C. Scope and limitations · VI. CONCLUSION · A. Outlook

The paper presents aria as a dependency-free, quantized runtime for on-device Stable Audio 3, while finding that validated steering control is genuine but narrowly scoped. Its outlook targets kernel fusion, attention optimization, and persistent serving to further improve deployment efficiency.

  • C. Scope and limitations: The systems contributions—runtime, quantization study, and efficiency comparison—rest on direct measurement and stand independently.The conclusion frames these contributions separately from the bounded steering case study.
  • C. Scope and limitations: Steering claims genuine control only for three attributes confirmed by independent checks, treating remaining attributes as negative results.The operating points replicate on held-out prompts and survive an audio-anchored retest.
  • VI. CONCLUSION: Aria is a dependency-free C/CUDA runtime that executes Stable Audio 3’s full text-to-music pipeline without a framework or third-party dependencies.The implementation is described as roughly 7.7k lines of C and runs on commodity CPUs and inexpensive GPUs.
  • VI. CONCLUSION: 8-bit weights keep prompt adherence, distributional quality, and taste within re-seed noise while 8-bit arithmetic is fastest on the GPU.Precision is treated as a deployment axis rather than an added-memory technique.
  • VI. CONCLUSION: 4-bit compression places the 1.2 B model on an 8 GB Pi after releasing the full-precision weights once compressed.This result demonstrates deployment on a memory-constrained Raspberry Pi-class device.
  • VI. CONCLUSION: The bit-exact activation-steering interface reproduces the reference dose–response, but sonic seasoning provides taste control only within a narrow window for some attributes.The steering study intervenes on a single model family using learned, distributional, oracles.
  • A. Outlook: Kernel fusion reverses the medium-tier long-form gap, enabling aria to overtake the official path there at exact precision.The remaining small-tier distance is attributed to full transformer self-attention, whose weights are too spread out for a banded approximation.
  • A. Outlook: A fused FlashAttention-class kernel behind a build flag and persistent server mode are identified as open directions for remaining performance and serving needs.Persistent serving would keep the model resident for multi-client serving.
Loading 2607.08526v1…