PaperlayerView exact revision on arXiv

Source-linked AI summary

ArtifactNet: Detecting AI-Generated Music via Forensic Residual Physics

Heewon Oh

arXiv:2604.16254v2cs.SDeess.AS

TL;DR

AI-music detectors often degrade on unseen generators and codec conditions. ArtifactNet addresses this by extracting codec-related forensic residuals with a bounded-mask UNet, decomposing them into HPSS features, and classifying them with a compact CNN. On an unseen ArtifactBench test partition, it achieves strong performance relative to larger baselines, while codec-aware training substantially reduces cross-codec drift.

  • Problem

    Existing detectors can achieve high in-distribution accuracy yet degrade on unseen generators, with compression codecs and genre distribution remaining confounders.

  • Method

    ArtifactNet extracts forensic residuals from STFT magnitudes with a bounded-mask UNet, converts them into seven-channel HPSS features, and classifies them using a compact CNN.

  • Results

    F1 = 0.9829 on the unseen ArtifactBench test partition, versus 0.7576 for CLAM and 0.7713 for SpecTTTra, while codec-aware training reduces cross-codec probability drift by 83%.

  • Takeaways & Limitations

    Forensic residual extraction provides a compact detection paradigm that transfers across the evaluated generator families and real-music sources.

  • Takeaways & Limitations

    The evaluation does not establish performance for future generators that avoid discrete codec quantization, and hard-negative refinement is not pure zero-shot generalization.

Abstract

from arXiv · show

We present ArtifactNet, a lightweight framework that detects AI-generated music by reframing the problem as forensic physics -- extracting and analyzing the physical artifacts that neural audio codecs inevitably imprint on generated audio. A bounded-mask UNet (ArtifactUNet, 3.6M parameters) extracts codec residuals from magnitude spectrograms, which are then decomposed via HPSS into 7-channel forensic features for classification by a compact CNN (0.4M parameters; 4.0M total). We introduce ArtifactBench, a multi-generator evaluation benchmark comprising 6,183 tracks (4,383 AI from 22 generators and 1,800 real from 6 diverse sources). Each track is tagged with bench_origin for fair zero-shot evaluation. On the unseen test partition (n=2,263), ArtifactNet achieves F1 = 0.9829 with FPR = 1.49%, compared to CLAM (F1 = 0.7576, FPR = 69.26%) and SpecTTTra (F1 = 0.7713, FPR = 19.43%) evaluated under identical conditions with published checkpoints. Codec-aware training (4-way WAV/MP3/AAC/Opus augmentation) further reduces cross-codec probability drift by 83% (Delta = 0.95 -> 0.16), resolving the primary codec-invariance failure mode. These results establish forensic physics -- direct extraction of codec-level artifacts -- as a more generalizable and parameter-efficient paradigm for AI music detection than representation learning, using 49x fewer parameters than CLAM and 4.8x fewer than SpecTTTra.

I. INTRODUCTION

AI-music detectors often lose performance on unseen generators, motivating methods based on generation artifacts rather than surface-level patterns. ArtifactNet tests a forensic-residual hypothesis grounded in neural audio codec quantization and evaluates it across diverse generators and real sources.

  • Motivation: Existing representation-learning and autoencoder-fingerprinting methods achieve high in-distribution accuracy but degrade on unseen generators.Compression codecs and genre distribution remain unresolved confounders.
  • Prior limitations: 99.8% accuracy from autoencoder reconstruction-error classifiers still coexists with structural generalization failure on unseen decoders.The prior work explicitly asks whether AI-generated music can be identified regardless of content.
  • Forensic hypothesis: Neural audio codecs use residual vector quantization, whose irreversible quantization gap can produce residual structures that differ between AI-generated and human recordings.The paper terms this phenomenon forensic residual amplification.
  • Scope: The paper claims residual asymmetry is a stable, learnable discriminator only within the evaluated 22 generators and 6 real-music sources.It explicitly does not claim RVQ loss is the sole causal source or that future generators cannot alter the balance.
  • Contributions: ArtifactNet reframes detection as amplifying and characterizing source-separation residuals rather than classifying learned embeddings.Its contributions include ArtifactUNet, HPSS-based forensic features, ArtifactBench, and a public SONICS evaluation.

C. Audio Forensics and Source Separation

ArtifactNet repurposes source separation and HPSS as forensic amplifiers, using a bounded-mask UNet to extract residuals before lightweight CNN classification. Codec-aware training suppresses compression artifacts that otherwise confound the residual signal.

  • Forensic repurposing: The method repurposes Demucs-style source separation and HPSS for forensic residual analysis rather than their conventional audio-analysis purposes.This bridges audio forensics with source separation and targets generation artifacts instead of content.
  • Pipeline: ArtifactNet processes audio through residual extraction, 7-channel HPSS feature computation, and lightweight CNN classification in a 4.0M-parameter pipeline.The pipeline uses ArtifactUNet as its residual-extraction front end.
  • ArtifactUNet: ArtifactUNet predicts a multiplicative mask r = m⊙X on the input STFT magnitude to extract forensic residuals.The bounded-mask design constrains m to [0, 0.5], discouraging passage of the entire signal.
  • Training and architecture: ArtifactUNet uses a 3.6M-parameter U-Net with knowledge-distillation training followed by classifier steering and codec-aware fine-tuning.The architecture includes a GatedResidualBlock bottleneck and requires approximately 16 MB on disk.
  • Codec invariance: Codec-aware training exposes the model to WAV, MP3, AAC, and Opus variants so it suppresses codec-induced artifacts rather than encoding them as forensic evidence.Without this regime, MP3 can flip real music to AI-positive and AAC can flip AI tracks to real-negative.

C. 7-Channel Forensic Features

The forensic representation decomposes extracted residuals into harmonic, percussive, temporal, and spectral channels before classification. This applies HPSS to residuals as a forensic signal rather than only as a music-analysis tool.

  • Feature construction: HPSS decomposes the extracted residual magnitude spectrogram into harmonic and percussive components using median filtering along time and frequency axes.Temporal derivatives and spectral statistics expand the representation to seven channels.
  • Forensic interpretation: The harmonic component captures pitched content left unattributed to any source, while the percussive component captures leaked transient energy from rhythmic elements.The paper reports structurally larger harmonic residuals for AI-generated audio in its measurements.
  • Representation and classification: Table I summarizes the seven-channel forensic feature representation used by the classifier.The compact CNN processes four-second segments and aggregates segment probabilities by song-level median thresholding.

A. Evaluation Philosophy: OOD Taxonomy

ArtifactNet separates deployment shift into explicit OOD axes and reports evaluations with their relevant axes annotated. The protocol combines stratified training data, codec augmentation, and cross-benchmark checks while distinguishing ranking stability from unseen-generator generalization.

  • OOD taxonomy: The paper evaluates four distinct deployment-time OOD axes rather than treating OOD as a single undifferentiated concept.This decomposition clarifies which distribution shift each result exercises.
  • Reporting protocol: Performance numbers are annotated with the OOD axes exercised by each evaluation.Table II catalogs the evaluated axes.
  • Training design: Phase 3 training applies four-way codec augmentation to every sample, while Phase 2 and the CNN use stratified sampling across 28 source subsets.The regime intentionally includes compressed real music to address a Phase 2 false-positive failure mode.
  • Cross-benchmark evaluation: The SONICS evaluation uses the full public test split under unified preprocessing, thresholding, and published baseline inference code without retraining.Its purpose is to test cross-benchmark ranking stability, not to establish OOD-generator performance.
  • Ranking stability: ArtifactNet’s ranking over SpecTTTra and CLAM is preserved on SONICS, reducing concern that ArtifactBench’s construction alone explains the ordering.The interpretation is explicitly limited to a uniform protocol across both benchmarks.
  • Scope boundary: SONICS is limited to five generator families that overlap ArtifactNet’s training mix, so it does not test generalization across unseen generators.The paper assigns unseen-generator claims to the separate ArtifactBench evaluation.

D. ArtifactBench: Multi-Generator Fair Evaluation

ArtifactBench extends fragmented evaluation axes into a reproducible benchmark for fair zero-shot comparison. Its held-out test partition covers diverse generators and real-music sources unseen by the evaluated models.

  • Benchmark motivation: ArtifactBench was constructed to improve reproducibility because exact reconstruction of SONICS and MoM real partitions is unstable.It is intended to unify and extend generator diversity, real-music diversity, hard negatives, codec stress, and generator drift.
  • Benchmark positioning: ArtifactBench is designed as a reproducible supplement rather than a replacement for SONICS or MoM.Its purpose is to consolidate evaluation dimensions that remain fragmented across those benchmarks.
  • Benchmark contents: 6,183 tracks comprise 4,383 AI-generated tracks from 22 generators and 1,800 real tracks from 6 diverse sources.
  • Evaluation protocol: The benchmark marks each track with bench_origin and restricts fair zero-shot evaluation to the test partition, containing 2,263 tracks.The test partition contains 1,388 AI and 875 real tracks and is unseen by all three models during training.

1) Observed result:

Under identical conditions on ArtifactBench’s unseen test partition, ArtifactNet ranks first and uses substantially fewer parameters than the baselines. The models exhibit distinct error patterns across source subsets.

  • Observed result: ArtifactNet achieves F1 = 0.9829 versus 0.7576 for CLAM and 0.7713 for SpecTTTra.All models use the same files, preprocessing, threshold τ = 0.5, and unseen test partition.
  • Observed result: ArtifactNet uses 49× fewer parameters than CLAM and 4.8× fewer than SpecTTTra.
  • Asymmetric failure modes: CLAM has high recall but catastrophic precision collapse, whereas SpecTTTra has high precision but low recall on out-of-distribution generators.The reported grouped values are CLAM recall 87.6% and precision 66.7%, versus SpecTTTra precision 85.2% and recall 70.5%.
  • Asymmetric failure modes: CLAM’s 69.3% real-music FPR reflects systematic false-positive bias, while SpecTTTra’s 70.5% recall reflects missed generators outside its training distribution.

E. Codec Robustness

ArtifactNet retains forensic signal after single-pass Demucs laundering, but the signal degrades. The analysis also reports codec-related robustness and a scope boundary around laundering attacks.

  • Codec robustness: Detection performance is stable within ±1.1 pp across MP3, Opus, and the tested codec conditions.The evaluated bitrate ranges are MP3 128–320 kbps and Opus 128–192 kbps.
  • Demucs laundering: Single-pass Demucs laundering reduces F1 from 0.9950 to 0.9592, while laundered AI-versus-real AUC remains 0.9651.
  • Demucs laundering: The forensic signal degrades but is not erased under single-pass laundering.
  • Scope: The laundering experiment does not evaluate multi-pass or adaptive laundering.Augmented retraining with Demucs-separated examples is included in ArtifactNet’s training data.

H. UNet Codec-Aware Training Ablation

The Phase 2 UNet is strongly codec-sensitive, with compression shifting predictions in opposite directions for real and AI tracks. Four-way codec-aware training substantially reduces this drift.

  • Phase 2 results: For real music, Phase 2 cross-codec probability delta is ∆ = 0.95, with MP3 producing 98.7% FPR; for AI tracks, ∆ = 0.72 and AAC creates false negatives.
  • Phase 3 results: Codec-aware training reduces real-track drift from ∆ = 0.95 to 0.16 (−83%) and AI-track drift from ∆ = 0.72 to 0.14 (−81%).Training uses WAV, MP3 128 kbps, AAC 128 kbps, and Opus 128 kbps variants.
  • Phase 3 results: The codec-aware UNet suppresses codec-induced spectral artifacts in residuals and prevents MP3 encoding from flipping real music to AI-positive.
  • Operating threshold: At FPR ≤5%, corresponding to τ ≈0.28, ArtifactNet achieves TPR = 99.1%.F1 remains above 0.98 across thresholds τ ∈[0.05, 0.9].

V. DISCUSSION

ArtifactNet’s compact forensic approach preserves its advantage across evaluated benchmarks and supports rapid adaptation, while its claims remain bounded by the evaluation protocol and training assumptions.

  • A 4.0M-parameter system outperforms CLAM and SpecTTTra by F1 margins of +0.23 and +0.21 on ArtifactBench, with the same ranking on SONICS.ArtifactNet’s aggregate FPR on real music is 1.5%, compared with CLAM’s 69.3%.
  • ArtifactNet targets codec-related forensic properties rather than generator- or style-dependent learned representations.The approach uses source-separation residuals and forensic feature processing to focus on artifacts associated with neural-codec generation.
  • A compact detector can be rapidly adapted to new failure modes by collecting hard negatives and fine-tuning a 4M-parameter network.The paper frames this as an asymmetric defender advantage rather than permanent generalization across all future audio distributions.
  • Hard-negative refinement expands training toward false-positive regions, so it is an error-driven robustness refinement rather than pure zero-shot generalization.The remaining failure modes can be absorbed without redesigning the system, within the paper’s stated scope.

B. Limitations

ArtifactNet’s forensic signal is constrained by input bandwidth, difficult codec artifacts, certain generators and segments, untested laundering strategies, and evolving generator families. ArtifactBench is therefore a point-in-time evaluation whose scope depends on shared neural-codec signatures.

  • 44.1 kHz residuals are required for high-frequency RVQ artifacts, making lossless or high-quality input a practical constraint.Reduced sample rates such as 16 kHz attenuate the forensic signal.
  • 98.7% FPR on low-quality MP3 archives exposed a major hard-negative failure, while codec-aware training reduced FPR to 8.0% on the FMA subset.The FMA hard-negative subset contains 1,350 MP3 VBR tracks.
  • Latest-generation Udio tracks reach 87% TPR because quiet and transition segments resemble real music in harmonic-percussive ratios.Auxiliary harmonic-percussive ratio statistics are proposed as a possible improvement.
  • Single-pass Demucs laundering lowers TPR to 94.0%, while multi-pass, adaptive laundering, and aggressive remastering remain unevaluated.These untested laundering strategies may degrade performance further.
  • The method’s signature may not transfer to generator families that avoid discrete neural-codec quantization, such as continuous-valued diffusion decoders without RVQ.The study provides no direct evidence for or against that case.
  • ArtifactBench v1 is a point-in-time snapshot requiring versioned updates as new generator families and releases appear.The benchmark’s maintenance cadence must track generator-release rates.

APPENDIX A BASELINE REPRODUCTION PROTOCOL

The appendix specifies a reproducible three-model evaluation protocol, documents missing-track handling, and reports fairness checks alongside benchmark and feature-analysis procedures. It also records an incomplete channel-ablation analysis and a bounded-mask diagnostic.

  • Reproduction protocol: The released protocol standardizes model inputs, excerpts, thresholds, and normalization while using published checkpoints for CLAM and SpecTTTra.ArtifactNet uses 44.1 kHz, 4-second chunks, and median aggregation; all models use threshold τ = 0.5 and decoded-audio clipping.
  • Missing-track handling: CLAM inference failed on 11 SONICS real tracks because of missing mirror files or decode errors, not a model failure.The appendix reports both exclusion and imputation accountings to make the denominator choice explicit.
  • Missing-track handling: 0.9993 F1 and 0.09% FPR are ArtifactNet’s results when missing tracks are excluded from all three models.Under this accounting, SpecTTTra reaches F1 = 0.8874 / FPR = 17.97%, while CLAM reaches F1 = 0.7661 / FPR = 67.03%.
  • Missing-track handling: 67.16% FPR is CLAM’s result when the 11 missing tracks are imputed as misclassified real, with the ranking and qualitative conclusions unchanged.The imputation changes CLAM’s FPR by +0.10 percentage points, and the released runner emits both accountings.
  • Fairness checks: Fairness checks use identical audio files, the same ArtifactBench test partition and threshold, no ArtifactNet-specific baseline preprocessing, and no baseline hyperparameter search.The baseline settings are taken from the authors’ published inference code.
  • Ablations: The planned per-channel ablation runs were not yet complete, while the bounded-mask ablation tests whether an unbounded mask produces a pass-through solution.The unbounded mask converges near unity, yields residual energy above 95% of input energy, and substantially lowers F1; precise values were deferred.
  • Feature analysis: Effective bandwidth is defined as the smallest upper frequency containing over 95% of cumulative residual magnitude energy under a specified full-track STFT.The computation uses NFFT = 2048, hop = 512, and averaging over the full track; low-pass filtering and mastering are acknowledged confounders.
Loading 2604.16254v2…