Distinguishing AI-Generated Music from Edited Audio as a Hard-Negative Robustness Task

Alexandru-Stefan Morosanu; Valerian Cecan; Stefan-Daniel Achirei; Laura Erhan

Source-linked AI summary

Distinguishing AI-Generated Music from Edited Audio as a Hard-Negative Robustness Task

Alexandru-Stefan Morosanu, Valerian Cecan, Stefan-Daniel Achirei, Laura Erhan

arXiv:2608.14916v1Aug 14, 2026cs.SDcs.AIcs.LG

TL;DR

AI-music detectors are often tested against originals, leaving their robustness to edited non-AI audio uncertain. This paper trains a detector on anchor-song-based AI and edited variants and achieves 0.811 balanced accuracy, while edited audio remains the harder class.

Problem
The study asks whether detectors can distinguish AI-generated music from edited variants of the same songs, which may contain AI-like processing traces.
Method
The authors compile YouTube variants grouped by reference song, split by anchor song, and fine-tune a pretrained PaSST model on 10-second audio clips.
Results
0.811 balanced accuracy is achieved on the held-out test set, while edited audio remains a challenging negative class.
Takeaways & Limitations
The results show that edited processing can produce artifacts overlapping with evidence used for AI detection, complicating reliable separation of AI-generated and edited audio.
Takeaways & Limitations
The edited class is based on available metadata and manual labeling, so it is not a perfectly controlled editing category.

Abstract

from arXiv · show

AI-generated music detectors are commonly evaluated against original songs, but real-world uploads are often remixed, re-encoded, pitch-shifted, or otherwise edited. These edited versions form a difficult negative class: they are not generated by AI, yet they may introduce spectral artifacts that resemble synthetic audio fingerprints. We study this problem as a hard-negative robustness setting for AI-generated music detection, focusing on AI-generated and edited variants derived from the same anchor songs. We compile a YouTube-based dataset of AI, edited, and original variants, using the original tracks only as references, and train a binary AI versus edited detector. Audio is processed as 10-second clips and passed as raw waveforms to a pretrained PaSST spectrogram transformer. To reduce leakage, all splits are performed by anchor song. On the held-out test set, the final video-level system achieves 0.811 balanced accuracy. At clip level, AI-generated clips reach an F1-score of 0.836, while edited clips reach a lower F1-score of 0.720. The results suggest that AI-generated music retains detectable fingerprint-like spectral cues beyond ordinary editing, but the lower edited-class F1-score shows that these cues can still overlap with artifacts from edited audio. Grad-CAM visualizations are used to inspect whether high-confidence predictions rely on localized time-frequency regions.

1 Introduction

As music-generation systems become full-song production tools, AI-music detection must distinguish synthetic tracks from edited non-AI audio that can produce similar artifacts. This work frames that distinction as a hard-negative robustness problem and evaluates it with song-grouped data, a pretrained PaSST model, and Grad-CAM analysis.

Motivation: Full-song generation from minimal prompts creates a practical need to identify AI-generated music in online catalogues, streaming platforms, and user-generated repositories.Current systems can generate vocals, instrumentation, arrangement, and production style.
Problem: Edited audio is a hard negative because remixing, pitch shifting, speed changes, re-encoding, compression, and mastering can introduce artifacts despite being non-AI.These edits make a clean original-versus-synthetic evaluation less representative of real uploads.
Method: The study compiles YouTube variants grouped by reference song and keeps all versions from each song within one partition to prevent leakage.The dataset includes AI-generated, edited, and original variants, while originals serve as references for the split structure.
Method: The detection pipeline fine-tunes a pretrained PaSST spectrogram transformer on 10-second audio clips and evaluates both clip-level and video-level performance.The contribution explicitly tests whether spectro-temporal cues distinguish AI generation from ordinary editing and post-processing.
Results and analysis: Edited audio remains a significant challenge, as production effects can mimic generative artifacts and complicate high-confidence AI predictions.Grad-CAM maps are used to inspect the time-frequency regions driving those predictions.

2 Related Work

Prior work establishes AI-generated music detection as an emerging field whose main challenges include robustness to audio manipulation and generalization to unseen generators. This work addresses that gap by treating edited and processed music as a hard negative class and building on spectrogram-transformer methods.

AI-generated music detection: AI-generated music detection is emerging as a distinct research topic, with controlled studies reporting high scores but identifying manipulation robustness and unseen-generator generalization as open problems.The cited studies examine audio-based detection and collections of tracks from Suno and Udio.
Robustness evaluation: This work extends evaluation beyond clean AI-versus-original splits by treating edited and processed music as the negative class.The robustness setting includes edited variants rather than using editing only as an external post-evaluation transformation.
Datasets: Recent datasets broaden synthetic-music detection through full-song classification, text-to-music detection, attribution, and closed-set or open-set evaluation.SONICS targets commercial-platform generations, while FakeMusicCaps covers multiple generation models.
Detection cues: Synthetic-audio detectors can exploit generative traces, including neural vocoder artifacts and spectral peaks linked to architectural choices such as deconvolution modules.These findings support using time–frequency representations to expose synthetic evidence.
Hard negatives: Non-AI operations such as compression, remixing, equalization, pitch shifting, time stretching, and remastering can alter spectra and resemble synthetic evidence.This concern parallels the move toward in-the-wild spoofing conditions in audio deepfake benchmarks.
Model foundation: PaSST follows spectrogram-transformer advances that use Patchout to reduce training cost while preserving strong audio-classification performance.The approach fine-tunes a pretrained PaSST model on raw audio clips with an internally computed log-Mel representation.

3 Data and Task

The study builds a YouTube dataset of related AI-generated, edited, and original song variants, then evaluates a binary AI-versus-edited task where edited audio serves as a hard negative. To limit song-specific leakage, all versions of each reference song remain within the same data partition.

Dataset construction: The dataset links multiple YouTube uploads to the same reference songs, collecting AI covers, remakes, remixes, slowed versions, and other edited variants rather than unrelated music.YouTube metadata and audio files were retrieved through the YouTube Data API, yt-dlp, ffmpeg, and ffprobe.
Dataset construction: 933 videos from 95 reference-song groups comprise the final subset, evenly divided among 311 AI-generated, 311 edited, and 311 original videos.The initial metadata file contained 3,640 labeled entries.
Task definition: The final classifier performs binary classification between ai and edited, while original videos remain reference material rather than an output class.Originals support comparison and prototype-based analysis but are not used as a target category.
Task definition: Edited audio is treated as a hard negative because compression, remixing, re-encoding, pitch shifting, time stretching, or mastering can create artifacts resembling AI evidence.The task tests whether AI-generated versions remain distinguishable from edited versions, not merely whether a song differs from its original.
Data splitting: All versions of each reference song are assigned to one partition, preventing models from learning song-specific cues and providing a cleaner estimate of generalization to unseen songs.The split uses reference-song groups rather than individual clips or videos, making the task harder and motivating balanced accuracy and macro F1-score reporting.

4 Methodology

The methodology processes 10-second mono waveform clips with a pretrained PaSST backbone that internally computes log-Mel features, then fine-tunes a binary AI-versus-edited classifier. Video-level decisions aggregate clip probabilities using a validation-selected fraction rule requiring sufficiently strong AI evidence in at least one third of clips.

Audio preprocessing: 10-second mono clips are resampled to 32 kHz and used as basic training and evaluation units, balancing local artifact detection with musical context.The model operates directly on waveforms rather than exported spectrogram images.
Model architecture: The pretrained PaSST model receives raw waveforms, computes 128-band log-Mel features internally, and predicts ai or edited classes.The frontend uses FFT size 1024, an 800-sample window, and a 320-sample hop, corresponding to a 10 ms temporal hop at 32 kHz.
Training: Training proceeds by first freezing the PaSST backbone and training only the classification head, then fine-tuning the full model end-to-end.The selected configuration uses batch size 4, head learning rate 10−3, fine-tuning learning rate 8 × 10−6, weight decay 8 × 10−5, and label smoothing 0.03.
Decision aggregation: Video-level evaluation compares mean, maximum, count-based, and fraction-based aggregation rules using clip-level AI probabilities.The selected rule classifies a video as AI-generated when at least 33% of clips satisfy p(ai) ≥0.6; otherwise, it is classified as edited.

5 Experiments and Results

Experiments evaluate PaSST on anchor-song-held-out 10-second clips and aggregated uploads, comparing it with handcrafted-feature and image-based baselines. PaSST improves separation in the edited-audio hard-negative setting, although edited clips remain a major source of confusion.

Evaluation setup: The held-out evaluation uses reference-song grouping, with 10-second clips and video-level predictions formed by aggregating clip probabilities per upload.The test set contains 1,659 AI-generated clips and 968 edited clips.
Baselines: 0.52 accuracy and 0.52 F1-score were achieved by Random Forest, indicating that handcrafted audio descriptors are insufficient for this distinction.The baseline uses descriptive global audio features rather than learned spectro-temporal representations.
Baselines: MobileNetV2 outperformed Random Forest but remained limited on balanced and macro metrics, motivating a move from generic image-based spectrogram classification to an audio transformer.MobileNetV2 was trained on exported spectrogram images and uses an image-pretrained convolutional network.
PaSST results: PaSST showed stronger performance on the AI class than the edited class, indicating useful generation-related cues while confirming edited audio as a difficult negative class.Edited audio can contain compression, re-encoding, pitch or tempo changes, and other processing traces that overlap with detector cues.
Video-level evaluation: Video-level aggregation produced higher scores than clip-level macro F1, suggesting that combining evidence across an upload reduces the influence of ambiguous or noisy clips.A single suspicious segment does not determine the whole-video label; the decision depends on the distribution of AI evidence across the upload.
Interpretation: The results indicate that PaSST can separate AI-generated and edited music variants to a useful degree in this specific hard-negative setting, not as a general-purpose detector for all AI-generated music.Edited audio remains a major source of confusion, while simpler descriptors and image-based spectrogram classification are weaker approaches.

6 Qualitative Analysis

Qualitative analysis compares same-song AI-generated and edited clips through log-Mel spectrograms and Grad-CAM maps. The visualizations show distinct but overlapping localized time-frequency evidence, supporting edited audio as a hard negative without implying a universal AI fingerprint.

Spectrogram representation: Log-Mel visualizations encode time-frequency energy, with brighter colors indicating higher energy and horizontal bands commonly corresponding to harmonic musical content.Vertical structures can indicate onsets, transients, abrupt edits, or other short-time signal changes.
Spectrogram comparison: Same-song log-Mel spectrograms show repeated harmonic bands and dense low-frequency energy in the AI clip, while the edited clip differs especially in mid and high frequencies.Both clips share musical content, making their comparison more meaningful; editing is therefore not a clean negative class.
Grad-CAM analysis: 0.909 p(ai) is assigned to the correctly classified AI-generated clip, whose Grad-CAM map highlights localized harmonic and transient-like regions.The localized activations suggest the model is not relying only on global loudness or average spectral energy.
Grad-CAM analysis: 0.887 p(edited) is assigned to the correctly classified edited clip, whose Grad-CAM map emphasizes different localized regions in the mid and upper Mel bins.These regions are consistent with localized spectral changes introduced by post-processing and editing.
Interpretation: The AI and edited activation maps are structured but not cleanly separable into distinct class-specific areas, indicating overlapping spectro-temporal evidence.This overlap matches the lower edited-class F1-score and supports treating edited audio as a hard negative.

7 Limitations

The study’s limitations concern uncertain class labeling, incomplete genre coverage, uncontrolled editing transformations, and limited baseline comparisons. These constraints may affect interpretation and generalization beyond the collected online variants.

Class labeling: Edited uploads cannot always be verified as purely human-edited, so the edited class represents a non-AI or non-fully-AI hard negative rather than a perfectly controlled category.Labels rely on available metadata and manual labeling.
Genre coverage: The dataset underrepresents styles including classical and purely instrumental music, so performance may vary across genres with different spectral and production characteristics.Collection depends on reference songs and variants available through the search procedure.
Editing control: Naturally occurring edits support realistic evaluation but do not isolate individual operations such as compression, pitch shifting, time stretching, or re-encoding.Controlled transformations would help measure which processing types most often produce AI-like evidence.
Baseline coverage: The evaluation includes only two lightweight reference baselines and omits stronger pretrained audio models and recent AI-audio detection methods.The reported baselines are Random Forest on handcrafted descriptors and MobileNetV2 on exported spectrogram images.

8 Conclusion

The paper frames edited audio as a realistic hard-negative class for AI-generated music detection and shows that pretrained spectrogram transformers can separate AI-generated from edited audio, though edited examples remain challenging. It argues that clean-original evaluations are overly optimistic and recommends broader robustness testing.

Hard-negative setting: The study treats edited and processed audio—not clean original music—as the negative class in a practical hard-negative detection setting.Such uploads may include compression, remixing, pitch or tempo changes, re-encoding, or other processing traces.
Dataset and evaluation: The main contribution is evaluating edited audio as a hard negative using a YouTube-based dataset of related AI-generated, edited, and original song versions with reference-based splitting.Original recordings serve as reference material, while reference song splitting reduces leakage between versions of the same song.
Interpretation and limitations: The results indicate that edited audio is not an easy negative because its processing traces can overlap with AI-like spectral evidence.The conclusion cautions that detection is not solved and that clean-original evaluations may be too optimistic.
Results: 0.811 balanced accuracy was achieved by the final video-level system on the held-out test set, alongside 0.817 accuracy and 0.813 macro F1-score.The system used audio clips and a pretrained PaSST spectrogram transformer.
Future work: Future work should test more generators, editing operations, and larger cross-platform datasets while examining detector decisions under controlled audio transformations.Edited audio is proposed as a robustness test because it exposes a realistic source of false AI evidence.