Source-linked AI summary
A Stem-Agnostic Approach to Hybrid AI Music Detection
Richa Namballa, François Rigaud, Romain Hennequin
TL;DR
Binary AI-music detectors struggle to localize generated sources in hybrid tracks mixed with human performances. This paper combines localized synthetic-content probabilities with target-stem energy masks in a stem-agnostic model; it performs best on high-frequency sources, while source separation limits accuracy.
Problem
Binary detectors struggle with hybrid human-AI tracks, and global detection does not reveal which specific components were generated.
Source: §1 · p. 1Method
A stem-agnostic model combines a time-frequency map of synthetic-content probabilities with the target stem’s Wiener energy mask.
Source: §1 · p. 1Results
Takeaways & Limitations
The framework provides a scalable foundation for fine-grained, stem-level AI-music detection.
Source: §6 · p. 4Performance is bounded by source-separation quality, with degraded bass and piano stems reducing detection performance compared with oracle stems.
Source: §6 · p. 4
Abstract
from arXiv · showhide
The inclusion of generative audio in the music production process has led to an increase in hybrid music tracks that blend authentic human performances with AI-generated stems, challenging traditional AI music detectors which operate in a binary setting. In this work, we propose a stem-agnostic framework for identifying synthetic audio sources within hybrid musical mixtures. We introduce the inspectrogram, a novel time-frequency representation that maps localized probabilities of synthetic content across the audio spectrum. By combining the inspectrogram with a Wiener filter estimating target stem energy dominance, a single CNN model evaluates whether the specific stem is generated. Trained on rendered hybrid mixtures and evaluated across various stem classes, our model achieves strong performance on high-frequency sources such as vocals, drums, and guitar, but struggles on the low-frequency, narrow-band bass. We conclude that the quality of separation impacts the detection accuracy and identify source separation as a primary bottleneck and a crucial direction for future research.
1. INTRODUCTION
Hybrid tracks challenge binary AI-music detectors because synthetic artifacts become less salient when mixed with human performances, and global predictions do not identify which components are generated. The paper addresses this with a stem-agnostic approach to locating synthetic sources within a mix.
1. INTRODUCTION
- Reliable AI-music detection matters to platforms and creators for content labeling, recommendation exclusion, and demonetizing fraudulent streams.The passage reports that platforms say AI-generated audio makes up the majority of daily new music uploads.Source: §1 · p. 1
- Hybrid tracks challenge binary detectors because generated artifacts become less salient in human performances, while global predictions do not identify which components are synthetic.The authors describe hybrid production as increasingly common as creators integrate generative tools into broader workflows.Source: §1 · p. 1
- The proposed stem-agnostic framework combines localized synthetic-content probabilities with a target stem’s Wiener filter to identify generated sources in hybrid tracks.Performance varies by stem: high-frequency sources fare better, while bass detection is constrained by source separation degrading critical spectral information.Source: §1 · p. 1
2. RELATED WORK
Existing audio detectors typically learn generation artifacts in a binary setting, while research on hybrid tracks has explored stem energy and frequency-dependent detection. This work builds on that foundation with detection localized in both time and frequency.
2. RELATED WORK
- Conventional detectors learn artifacts such as periodic spectral peaks but are generally trained to distinguish fully generated from entirely human-made tracks.Their generalization is also constrained because artifacts vary across generation platforms and codecs.Source: §2 · p. 1
- Rigaud et al. combine local binary detection outputs with target-stem SNR across frequency bands, finding that detection depends heavily on the generated stem’s relative energy.Their naive pipeline of source separation followed by binary detection performs poorly because separation fails to isolate artifacts with the generated stem.Source: §2 · p. 1
- De la Cruz et al. estimate the generated proportion of a track and report that drums and guitar are easier to detect than vocals and bass.This paper builds on with a more fine-grained approach using both time and frequency.Source: §2 · pp. 1–2
3. DATASETS AND FEATURES
The paper renders hybrid mixtures from existing music-separation corpora and constructs a localized synthetic-content map alongside target-stem energy masks. These aligned time-frequency representations provide the inputs for stem-level detection.
3.1. Hybrid Music Datasets
- The hybrid data uses EnCodec-rendered stems as a proxy for generation, pairing original and encoded versions to focus detection on synthetic artifacts.MUSDB18-HQ supplies all 16 original-or-encoded combinations for each of 150 tracks; MoisesDB provides an out-of-distribution test set with six standardized stems.Source: §3.1 · p. 2
3.2. Inspectrogram
- Band-Split Regression applies independent logistic classifiers to 32 frequency bands, producing localized synthetic-content probabilities that form the inspectrogram across sliding segments.The chosen configuration uses 5-second segments with a 0.25-second hop; performance is more reliable above 3 kHz and degrades at lower frequencies.Source: §3.2 · p. 2
3.3. Wiener Filter Masks
- The Wiener energy mask represents target-stem energy relative to the mixture, aligned to the inspectrogram’s time-frequency dimensions.The masks are computed for ground-truth oracle stems and for stems separated by HTDemucs; the figure illustrates synthetic probability rising when generated vocals are active.Source: §3.3 · pp. 2–3, Figure 2 · p. 2
4. HYBRID DETECTION MODEL
A single CNN evaluates a target stem using aligned inspectrogram and Wiener-mask inputs, with labels indicating whether that target is generated or real. Its use is limited by stem-specific separation quality, particularly for bass.
4. HYBRID DETECTION MODEL
- The CNN receives two-channel windows combining inspectrogram and energy-mask values across 64 temporal chunks and 32 frequency bands.Each window covers approximately 20 seconds, and windows are spaced by a 5-second hop.Source: §4 · p. 3
- A stem is labeled generated or real based on its energy mask, and window-level predictions are averaged into a track-level probability.This formulation assumes each target stem is entirely authentic or synthetic; hybrid target stems remain a limitation.Source: §4 · p. 3
- Training excludes windows with substantial low-energy mixture content or a virtually silent target stem.The model is trained on rendered hybrid MUSDB with an 80/10/10 track-level train, validation, and test split.Source: §4 · p. 3
- One stem-agnostic CNN can evaluate any mixture stem when its energy mask is available, rather than requiring a dedicated detector for each stem type.Processing multiple chunks also captures local temporal dependencies alongside spectral variation.Source: §4 · p. 3
- Bass is excluded from final target stems because HTDemucs discards higher-order harmonics and note transients, and retraining with separated masks still performs no better than chance.The reported evaluation instead targets vocals, drums, and “other,” along with unseen guitar and piano stems.Source: §4 · p. 3
5. RESULTS
The model detects vocals, drums, and other stems in MUSDB under both oracle and source-separated conditions, and its strong vocals-and-drums performance generalizes to MoisesDB. Piano predictions are less reliable, especially with separated or low-energy stems.
5. RESULTS
- Oracle-trained detection transfers to MSS test stems, while MSS-based training causes only minor degradation on drums and “other.”The authors suggest HTDemucs’s MUSDB training may help explain the oracle-based model’s performance on MSS energy masks.Source: §5 · p. 4
- The model detects all three tested stem types on MUSDB in both oracle and MSS settings, and strong vocals-and-drums detection generalizes to MoisesDB.The oracle-trained model also identifies isolated guitar and piano relatively accurately in MoisesDB’s oracle test set, despite training on those sources only within “other.”Source: §5 · p. 4
- Piano FPRs rise for MSS-based models and MSS-6, with separation difficulty, sparse activity, and low energy contributing to ambiguous predictions.The authors suggest post-processing based on overall stem energy could help assess prediction reliability.Source: §5 · p. 4
- In the MoisesDB example, predictions approach 0 or 1 when target-stem energy is high, but hover around 0.5 during low-energy or silent passages.Figure 5 plots per-window generated-stem probability alongside the maximum energy-mask value by stem for an MSS-6 example.Source: §5 · p. 4, Figure 5 · p. 4
6. CONCLUSION
The paper presents a framework for detecting synthetic stems in hybrid mixtures, with strongest performance on high-frequency sources. Its practical performance is bounded by source-separation quality, and the experiments use a single neural audio codec.
6. CONCLUSION
- The framework combines localized generation probabilities in an inspectrogram with target-stem energy masks to detect synthetic stems, performing best on vocals, drums, and guitar.The model associates source-energy dominance with synthetic-content likelihood.Source: §6 · p. 4
- Experiments use a single neural audio codec; adaptation to other codecs or commercial services remains a proposed extension, and the required tuning is unresolved.The authors conjecture that codec-specific inspectrogram training may allow reuse of the downstream CNN, but identify testing that adaptability as future research.Source: §6 · p. 4
- Detection performance and practicality are bounded by source-separation quality, with degraded bass and piano stems reducing performance versus the oracle setting.The authors propose exploring newer separation models or lightweight target-specific separators that predict low-resolution Wiener filters.Source: §6 · p. 4