PaperlayerView exact revision on arXiv

Source-linked AI summary

ArtifactNet: Detecting AI-Generated Music via Forensic Residual Physics

Heewon Oh

arXiv:2604.16254v2cs.SDeess.AS

TL;DR

AI-music detectors often degrade on unseen generators and codec conditions. ArtifactNet addresses this by extracting codec-related forensic residuals with a bounded-mask UNet, decomposing them into HPSS features, and classifying them with a compact CNN. On an unseen ArtifactBench test partition, it achieves strong performance relative to larger baselines, while codec-aware training substantially reduces cross-codec drift.

Abstract

from arXiv · show

We present ArtifactNet, a lightweight framework that detects AI-generated music by reframing the problem as forensic physics -- extracting and analyzing the physical artifacts that neural audio codecs inevitably imprint on generated audio. A bounded-mask UNet (ArtifactUNet, 3.6M parameters) extracts codec residuals from magnitude spectrograms, which are then decomposed via HPSS into 7-channel forensic features for classification by a compact CNN (0.4M parameters; 4.0M total). We introduce ArtifactBench, a multi-generator evaluation benchmark comprising 6,183 tracks (4,383 AI from 22 generators and 1,800 real from 6 diverse sources). Each track is tagged with bench_origin for fair zero-shot evaluation. On the unseen test partition (n=2,263), ArtifactNet achieves F1 = 0.9829 with FPR = 1.49%, compared to CLAM (F1 = 0.7576, FPR = 69.26%) and SpecTTTra (F1 = 0.7713, FPR = 19.43%) evaluated under identical conditions with published checkpoints. Codec-aware training (4-way WAV/MP3/AAC/Opus augmentation) further reduces cross-codec probability drift by 83% (Delta = 0.95 -> 0.16), resolving the primary codec-invariance failure mode. These results establish forensic physics -- direct extraction of codec-level artifacts -- as a more generalizable and parameter-efficient paradigm for AI music detection than representation learning, using 49x fewer parameters than CLAM and 4.8x fewer than SpecTTTra.

I. INTRODUCTION

AI-music detectors often lose performance on unseen generators, motivating methods based on generation artifacts rather than surface-level patterns. ArtifactNet tests a forensic-residual hypothesis grounded in neural audio codec quantization and evaluates it across diverse generators and real sources.

Motivation

  • Existing representation-learning and autoencoder-fingerprinting methods achieve high in-distribution accuracy but degrade on unseen generators.Compression codecs and genre distribution remain unresolved confounders.Source: §I · p. 1

Prior limitations

  • 99.8% accuracy from autoencoder reconstruction-error classifiers still coexists with structural generalization failure on unseen decoders.The prior work explicitly asks whether AI-generated music can be identified regardless of content.Source: §I · p. 1

Forensic hypothesis

  • Neural audio codecs use residual vector quantization, whose irreversible quantization gap can produce residual structures that differ between AI-generated and human recordings.The paper terms this phenomenon forensic residual amplification.Source: §I · p. 1, B. Neural Codec Bottleneck · p. 2

Scope

  • The paper claims residual asymmetry is a stable, learnable discriminator only within the evaluated 22 generators and 6 real-music sources.It explicitly does not claim RVQ loss is the sole causal source or that future generators cannot alter the balance.Source: §I · p. 1

Contributions

  • ArtifactNet reframes detection as amplifying and characterizing source-separation residuals rather than classifying learned embeddings.Its contributions include ArtifactUNet, HPSS-based forensic features, ArtifactBench, and a public SONICS evaluation.Source: §I · pp. 1–2

C. Audio Forensics and Source Separation

ArtifactNet repurposes source separation and HPSS as forensic amplifiers, using a bounded-mask UNet to extract residuals before lightweight CNN classification. Codec-aware training suppresses compression artifacts that otherwise confound the residual signal.

Forensic repurposing

Pipeline

  • ArtifactNet processes audio through residual extraction, 7-channel HPSS feature computation, and lightweight CNN classification in a 4.0M-parameter pipeline.The pipeline uses ArtifactUNet as its residual-extraction front end.Source: A. Architecture Overview · p. 2, Figure 1 · p. 2

ArtifactUNet

Training and architecture

  • ArtifactUNet uses a 3.6M-parameter U-Net with knowledge-distillation training followed by classifier steering and codec-aware fine-tuning.The architecture includes a GatedResidualBlock bottleneck and requires approximately 16 MB on disk.Source: B. ArtifactUNet: Bounded Mask Residual… · p. 3

Codec invariance

  • Codec-aware training exposes the model to WAV, MP3, AAC, and Opus variants so it suppresses codec-induced artifacts rather than encoding them as forensic evidence.Without this regime, MP3 can flip real music to AI-positive and AAC can flip AI tracks to real-negative.Source: B. ArtifactUNet: Bounded Mask Residual… · p. 3

C. 7-Channel Forensic Features

The forensic representation decomposes extracted residuals into harmonic, percussive, temporal, and spectral channels before classification. This applies HPSS to residuals as a forensic signal rather than only as a music-analysis tool.

Feature construction

  • HPSS decomposes the extracted residual magnitude spectrogram into harmonic and percussive components using median filtering along time and frequency axes.Temporal derivatives and spectral statistics expand the representation to seven channels.Source: C. 7-Channel Forensic Features · p. 3

Forensic interpretation

  • The harmonic component captures pitched content left unattributed to any source, while the percussive component captures leaked transient energy from rhythmic elements.The paper reports structurally larger harmonic residuals for AI-generated audio in its measurements.Source: C. 7-Channel Forensic Features · p. 3

Representation and classification

A. Evaluation Philosophy: OOD Taxonomy

ArtifactNet separates deployment shift into explicit OOD axes and reports evaluations with their relevant axes annotated. The protocol combines stratified training data, codec augmentation, and cross-benchmark checks while distinguishing ranking stability from unseen-generator generalization.

OOD taxonomy

  • The paper evaluates four distinct deployment-time OOD axes rather than treating OOD as a single undifferentiated concept.This decomposition clarifies which distribution shift each result exercises.Source: A. Evaluation Philosophy: OOD Taxonomy · pp. 3–4

Reporting protocol

Training design

  • Phase 3 training applies four-way codec augmentation to every sample, while Phase 2 and the CNN use stratified sampling across 28 source subsets.The regime intentionally includes compressed real music to address a Phase 2 false-positive failure mode.Source: B. Training Data · p. 4, Table III · p. 4

Cross-benchmark evaluation

  • The SONICS evaluation uses the full public test split under unified preprocessing, thresholding, and published baseline inference code without retraining.Its purpose is to test cross-benchmark ranking stability, not to establish OOD-generator performance.Source: C. SONICS Full-Test 3-Way Evaluation… · p. 4, Table IV · p. 4

Ranking stability

  • ArtifactNet’s ranking over SpecTTTra and CLAM is preserved on SONICS, reducing concern that ArtifactBench’s construction alone explains the ordering.The interpretation is explicitly limited to a uniform protocol across both benchmarks.Source: C. SONICS Full-Test 3-Way Evaluation… · p. 4

Scope boundary

  • SONICS is limited to five generator families that overlap ArtifactNet’s training mix, so it does not test generalization across unseen generators.The paper assigns unseen-generator claims to the separate ArtifactBench evaluation.Source: C. SONICS Full-Test 3-Way Evaluation… · p. 4

D. ArtifactBench: Multi-Generator Fair Evaluation

ArtifactBench extends fragmented evaluation axes into a reproducible benchmark for fair zero-shot comparison. Its held-out test partition covers diverse generators and real-music sources unseen by the evaluated models.

Benchmark motivation

  • ArtifactBench was constructed to improve reproducibility because exact reconstruction of SONICS and MoM real partitions is unstable.It is intended to unify and extend generator diversity, real-music diversity, hard negatives, codec stress, and generator drift.Source: D. ArtifactBench: Multi-Generator Fair… · pp. 4–5

Benchmark positioning

  • ArtifactBench is designed as a reproducible supplement rather than a replacement for SONICS or MoM.Its purpose is to consolidate evaluation dimensions that remain fragmented across those benchmarks.Source: D. ArtifactBench: Multi-Generator Fair… · p. 5

Benchmark contents

Evaluation protocol

1) Observed result:

Under identical conditions on ArtifactBench’s unseen test partition, ArtifactNet ranks first and uses substantially fewer parameters than the baselines. The models exhibit distinct error patterns across source subsets.

Observed result

  • ArtifactNet achieves F1 = 0.9829 versus 0.7576 for CLAM and 0.7713 for SpecTTTra.All models use the same files, preprocessing, threshold τ = 0.5, and unseen test partition.Source: 1) Observed result · pp. 5–6
  • ArtifactNet uses 49× fewer parameters than CLAM and 4.8× fewer than SpecTTTra.Source: 1) Observed result · p. 6

Asymmetric failure modes

E. Codec Robustness

ArtifactNet retains forensic signal after single-pass Demucs laundering, but the signal degrades. The analysis also reports codec-related robustness and a scope boundary around laundering attacks.

Codec robustness

  • Detection performance is stable within ±1.1 pp across MP3, Opus, and the tested codec conditions.The evaluated bitrate ranges are MP3 128–320 kbps and Opus 128–192 kbps.Source: E. Codec Robustness · p. 6

Demucs laundering

Scope

  • The laundering experiment does not evaluate multi-pass or adaptive laundering.Augmented retraining with Demucs-separated examples is included in ArtifactNet’s training data.Source: G. Adversarial Robustness: Demucs… · p. 7

H. UNet Codec-Aware Training Ablation

The Phase 2 UNet is strongly codec-sensitive, with compression shifting predictions in opposite directions for real and AI tracks. Four-way codec-aware training substantially reduces this drift.

Phase 2 results

Phase 3 results

  • Codec-aware training reduces real-track drift from ∆ = 0.95 to 0.16 (−83%) and AI-track drift from ∆ = 0.72 to 0.14 (−81%).Training uses WAV, MP3 128 kbps, AAC 128 kbps, and Opus 128 kbps variants.Source: H. UNet Codec-Aware Training Ablation · p. 7
  • The codec-aware UNet suppresses codec-induced spectral artifacts in residuals and prevents MP3 encoding from flipping real music to AI-positive.Source: H. UNet Codec-Aware Training Ablation · p. 7

Operating threshold

  • At FPR ≤5%, corresponding to τ ≈0.28, ArtifactNet achieves TPR = 99.1%.F1 remains above 0.98 across thresholds τ ∈[0.05, 0.9].Source: §I · p. 7

V. DISCUSSION

ArtifactNet’s compact forensic approach preserves its advantage across evaluated benchmarks and supports rapid adaptation, while its claims remain bounded by the evaluation protocol and training assumptions.

  • A 4.0M-parameter system outperforms CLAM and SpecTTTra by F1 margins of +0.23 and +0.21 on ArtifactBench, with the same ranking on SONICS.ArtifactNet’s aggregate FPR on real music is 1.5%, compared with CLAM’s 69.3%.Source: A. Why Forensic Residual Extraction… · p. 7
  • ArtifactNet targets codec-related forensic properties rather than generator- or style-dependent learned representations.The approach uses source-separation residuals and forensic feature processing to focus on artifacts associated with neural-codec generation.Source: A. Why Forensic Residual Extraction… · pp. 7–8
  • A compact detector can be rapidly adapted to new failure modes by collecting hard negatives and fine-tuning a 4M-parameter network.The paper frames this as an asymmetric defender advantage rather than permanent generalization across all future audio distributions.Source: A. Why Forensic Residual Extraction… · p. 8
  • Hard-negative refinement expands training toward false-positive regions, so it is an error-driven robustness refinement rather than pure zero-shot generalization.The remaining failure modes can be absorbed without redesigning the system, within the paper’s stated scope.Source: A. Why Forensic Residual Extraction… · p. 8

B. Limitations

ArtifactNet’s forensic signal is constrained by input bandwidth, difficult codec artifacts, certain generators and segments, untested laundering strategies, and evolving generator families. ArtifactBench is therefore a point-in-time evaluation whose scope depends on shared neural-codec signatures.

  • 44.1 kHz residuals are required for high-frequency RVQ artifacts, making lossless or high-quality input a practical constraint.Reduced sample rates such as 16 kHz attenuate the forensic signal.Source: B. Limitations · p. 8
  • 98.7% FPR on low-quality MP3 archives exposed a major hard-negative failure, while codec-aware training reduced FPR to 8.0% on the FMA subset.The FMA hard-negative subset contains 1,350 MP3 VBR tracks.Source: B. Limitations · p. 8
  • Latest-generation Udio tracks reach 87% TPR because quiet and transition segments resemble real music in harmonic-percussive ratios.Auxiliary harmonic-percussive ratio statistics are proposed as a possible improvement.Source: B. Limitations · p. 8
  • Single-pass Demucs laundering lowers TPR to 94.0%, while multi-pass, adaptive laundering, and aggressive remastering remain unevaluated.These untested laundering strategies may degrade performance further.Source: B. Limitations · p. 8
  • The method’s signature may not transfer to generator families that avoid discrete neural-codec quantization, such as continuous-valued diffusion decoders without RVQ.The study provides no direct evidence for or against that case.Source: B. Limitations · p. 8
  • ArtifactBench v1 is a point-in-time snapshot requiring versioned updates as new generator families and releases appear.The benchmark’s maintenance cadence must track generator-release rates.Source: B. Limitations · p. 9

APPENDIX A BASELINE REPRODUCTION PROTOCOL

The appendix specifies a reproducible three-model evaluation protocol, documents missing-track handling, and reports fairness checks alongside benchmark and feature-analysis procedures. It also records an incomplete channel-ablation analysis and a bounded-mask diagnostic.

Reproduction protocol

  • The released protocol standardizes model inputs, excerpts, thresholds, and normalization while using published checkpoints for CLAM and SpecTTTra.ArtifactNet uses 44.1 kHz, 4-second chunks, and median aggregation; all models use threshold τ = 0.5 and decoded-audio clipping.Source: Appendix A · p. 9

Missing-track handling

  • CLAM inference failed on 11 SONICS real tracks because of missing mirror files or decode errors, not a model failure.The appendix reports both exclusion and imputation accountings to make the denominator choice explicit.Source: Appendix A · p. 9
  • 0.9993 F1 and 0.09% FPR are ArtifactNet’s results when missing tracks are excluded from all three models.Under this accounting, SpecTTTra reaches F1 = 0.8874 / FPR = 17.97%, while CLAM reaches F1 = 0.7661 / FPR = 67.03%.Source: Appendix A · p. 9
  • 67.16% FPR is CLAM’s result when the 11 missing tracks are imputed as misclassified real, with the ranking and qualitative conclusions unchanged.The imputation changes CLAM’s FPR by +0.10 percentage points, and the released runner emits both accountings.Source: Appendix A · p. 9

Fairness checks

  • Fairness checks use identical audio files, the same ArtifactBench test partition and threshold, no ArtifactNet-specific baseline preprocessing, and no baseline hyperparameter search.The baseline settings are taken from the authors’ published inference code.Source: Appendix A · p. 9

Ablations

  • The planned per-channel ablation runs were not yet complete, while the bounded-mask ablation tests whether an unbounded mask produces a pass-through solution.The unbounded mask converges near unity, yields residual energy above 95% of input energy, and substantially lowers F1; precise values were deferred.Source: Appendix B · pp. 9–10

Feature analysis

  • Effective bandwidth is defined as the smallest upper frequency containing over 95% of cumulative residual magnitude energy under a specified full-track STFT.The computation uses NFFT = 2048, hop = 512, and averaging over the full track; low-pass filtering and mastering are acknowledged confounders.Source: Appendix B · p. 10

Related papers

Loading 2604.16254v2…