PaperlayerView exact revision on arXiv

Source-linked AI summary

Clean Accuracy Does Not Guarantee Provenance Robustness: A Prospective Codec-Stress Evaluation of Audio Attribution

Gang Shi

arXiv:2609.07981v1cs.SDcs.LG

TL;DR

Clean attribution scores do not answer how much performance survives codec transport, even though analysts often receive transcoded audio. This paper prospectively measures closed-set attribution after single-stage transport with an analysis region fixed before attribution training; substantial, condition-dependent losses are observed on both corpora. The findings are bounded to the tested tasks, corpora, representations, and codec grid.

  • Problem

    Clean benchmarks report near-ceiling synthetic-speech attribution, but the amount of performance retained after realistic codec transport remains a measurement question.

    Source: §I · p. 1

  • Method

    The study prospectively evaluates closed-set attribution after codec transport using an analysis region fixed from fidelity metadata before any attribution model was trained.

    Source: §XII · p. 11

  • Results

    Codec transport removed a large fraction of attribution performance on both corpora, with losses varying substantially by condition and representation.

    Source: §I · p. 1, §XII · p. 11

  • Takeaways & Limitations

    A clean accuracy figure alone does not characterize deployment robustness, and the tested fidelity measures do not provide one common ordering of codec severity.

    Source: §X · p. 10, §XII · p. 11

    The evidence is bounded to two corpora, three representations, one single-stage codec grid, and closed-set tasks.

    Source: §XI · p. 10

Abstract

from arXiv · show

Audio provenance attribution - which system produced a synthetic utterance - is reported at near-ceiling accuracy on clean benchmarks, yet audio reaching an analyst has usually been transcoded. We report a prospectively registered measurement of closed-set attribution after single-stage codec transport, with the analysis region fixed from fidelity metadata before any attribution model was trained. On two corpora, in-support losses reach 53.5 [43.5, 63.6] and 70.3 [63.0, 77.5] Macro-F1 points for WavLM-Base+, and 61.0 [56.8, 65.1] and 49.8 [41.6, 57.9] for W2V2-BERT 2.0, under simultaneous component-level bands. Degradation is strongly condition- and representation-dependent: within one in-support grid WavLM losses run from -0.4 to +53.5 points, and the two encoders differ beyond a prespecified +/-5-point margin at six of twelve conditions. A clean-qualified ECAPA-TDNN and a Proxy-Anchor head degrade comparably, so the effect is not confined to one representation family or a weak linear head. The registered matched-fidelity comparison was not estimable on this grid, and waveform and perceptual measures order the conditions differently: MP3 at 8 kbit/s ranks mid-grid on SI-SDR but last on PESQ-WB while causing the largest loss. For the tested tasks, corpora, representations and codec grid, a clean accuracy figure does not by itself characterise deployment robustness.

I. INTRODUCTION

Clean-benchmark attribution scores leave unanswered how much performance survives realistic codec transport, a measurement needed before deployment. The paper places that question alongside detection and codec-attribution research, distinguishing transport stress from using a codec as the provenance signal.

I. INTRODUCTION

  • The paper asks how much clean attribution performance survives realistic transport and whether clean scores alone predict that amount.Audio reaching analysts has often passed through telephony, messaging, social platforms, or archival storage and may be compressed.Source: §I · p. 1
  • All three tested representations exceed 0.85 clean Macro-F1, yet losses reach 53.5 and 70.3 Macro-F1 points at one in-grid operating point.The contribution is bounded to the tested closed-set tasks, two corpora, three representations, and single-stage codec grid.Source: §I · p. 1
  • Within one in-support grid, degradation ranges from indistinguishable from zero to above 50 points, and encoder differences exceed the ±5-point margin at six of twelve conditions.The encoder ordering reverses across corpora, so one clean score or representation does not characterize transport robustness.Source: §I · p. 1
  • The registered matched-fidelity comparison was not estimable, while waveform and perceptual measures rank codec conditions differently.MP3 at 8 kbit/s sits mid-grid on SI-SDR but last on PESQ-WB.Source: §I · p. 1

II. RELATED WORK

  • The authors' literature audit finds codec-robustness measurements in detection, but not a comparable account of multi-class attribution under transport.The cited detection results concern binary detection and do not answer how grouped-source attribution performs across codec families and operating points.Source: §II · p. 2
  • Prior neural-codec work treats codec identity or codec artifacts as forensic signals, whereas this study tests whether a subsequent transport codec disrupts an existing attribution task.The paper also notes that neural-codec artifact reliance has been criticized as overspecialized for synthetic-speech detection.Source: §II · p. 2

III. EVALUATION DESIGN AND EVIDENCE DISCIPLINE

The evaluation separates development from prospective evidence and fixes the support boundary before attribution outcomes. Its inference accounts for component dependence and repeated seeded splits, while the registered matched-fidelity comparison lacks sufficient support on the tested grid.

III. EVALUATION DESIGN AND EVIDENCE DISCIPLINE

  • Development hypotheses were tested on separate prospective data under a protocol fixed in advance; the two evidence sets were never pooled.Only prospective evidence supports Sections V–IX.Source: §III · p. 2
  • The matched-fidelity rule admitted only one pair per corpus, below its two-pair minimum, so the registered comparison was not estimable on this grid.A secondary conditional-association analysis followed a calibration-informed, post-registration support rule and was not treated as a headline result.Source: §III · p. 2
  • The support boundary was computed from fidelity metadata and committed before any attribution head was trained or outcome inspected.LA and ST-Codecfake-OOD passed their registered admission thresholds; MLAAD-unseen had an empty support box and was designated sensitivity-only in advance.Source: §III · p. 2
  • Inference uses independent components rather than clips, with component-level intervals despite substantially larger clip counts.Components group clips sharing registered source or speaker identifiers; Table I summarizes the design and effective inference units.Source: §III · pp. 2–3, Table I · p. 3
  • Five registered seeds produced only three unique split configurations per corpus, so duplicate configurations carry repeated weight in the primary aggregate.A post-hoc analysis weighting unique configurations equally changed no primary decision.Source: §III · p. 3
  • Uncertainty uses a component cluster bootstrap with 1,000 replicates and simultaneous max-|t| bands, while decision thresholds and margins were frozen before execution.Class-signature stratification preserves class composition when used; the registered Q1b primary instead bootstraps components without stratification.Source: §III · p. 3

IV. EXPERIMENTAL SETUP

The experiments test two distinct closed-set provenance tasks with frozen representations and registered codec conditions. Clean qualification precedes stress testing, and the LA scored task varies in size across seeds.

IV. EXPERIMENTAL SETUP

  • The LA task is 3-way in two seeded runs and 5-way in three, whereas ST-Codecfake-OOD is 4-way in every run.Macro-F1 averages LA runs with differing task sizes under the registered protocol.Source: §IV · p. 4
  • WavLM-Base+ and W2V2-BERT 2.0 are the deciding frozen encoders; ECAPA-TDNN is a descriptive generality probe.The setup also includes a linear classifier and a reduced-scale Proxy-Anchor metric-learning head.Source: §IV · p. 4
  • The condition grid includes MP3, Opus, EnCodec, and DAC operating points plus a bandwidth bottleneck, µ-law, and two noise controls.All clips are 16 kHz mono before transformation; neural codecs use a 16→24→16 kHz resampling path.Source: §IV · p. 4
  • ASVspoof 2019 LA tests TTS-system identity, while ST-Codecfake-OOD tests codec-source identity; their attribution magnitudes are not compared.The two corpora test whether transport degradation recurs within each task definition.Source: §IV · pp. 3–4
  • Degradation is defined as clean minus transformed true multiclass Macro-F1, in points, so positive values indicate loss.MLAAD-unseen fails every registered clean qualification gate and is reported only as external-domain sensitivity.Source: §IV · p. 4, Table II · p. 4

V. RESULT 1: CODEC TRANSPORT REMOVES A LARGE FRACTION OF ATTRIBUTION PERFORMANCE

All three encoders passed clean qualification, yet codec transport produced large in-support attribution losses on both confirmatory corpora. The registered decision was supported on both corpora.

V. RESULT 1: CODEC TRANSPORT REMOVES A LARGE FRACTION OF ATTRIBUTION PERFORMANCE

  • WavLM-Base+ losses reached +53.5 [+43.5, +63.6] on LA and +70.3 [+63.0, +77.5] on ST-Codecfake-OOD; W2V2-BERT 2.0 reached +61.0 [+56.8, +65.1] and +49.8 [+41.6, +57.9].Source: §V · p. 4
  • The registered decision was supported on both corpora: simultaneous lower bounds exceeded 20 points at five and seven LA conditions for WavLM and W2V2-BERT, and one and three ST conditions, respectively.Source: §V · pp. 4–5
  • ECAPA-TDNN had simultaneous lower bounds above 20 points at five LA conditions and four ST-Codecfake-OOD conditions, but its clean qualification on ST is weighting-sensitive.Source: §V · p. 5
  • Proxy-Anchor also degraded under stress, with +41.7 [+37.7, +45.6] points at EnCodec-6k on LA and +41.0 [+30.1, +51.9] on ST-Codecfake-OOD.Source: §V · p. 5
  • The single largest loss on each corpus occurred at a conventional codec operating point: Opus at 6 kbit/s on LA and MP3 at 8 kbit/s on ST-Codecfake-OOD.Source: §V · p. 6

VI. RESULT 2: DEGRADATION IS STRONGLY CONDITION- AND REPRESENTATION-DEPENDENT

Codec-stress losses vary substantially by operating condition, and encoder robustness differences are also condition- and corpus-dependent. Equal weighting changes the number of conditions showing material encoder interaction on ST-Codecfake-OOD but not LA.

VI. RESULT 2: DEGRADATION IS STRONGLY CONDITION- AND REPRESENTATION-DEPENDENT

  • WavLM-Base+ losses ranged from −0.4 [−2.4, +1.7] at DAC nq = 20 to +53.5 [+43.5, +63.6] at MP3-8 on LA, and from +2.6 [−1.0, +6.2] at Opus-16 to +70.3 [+63.0, +77.5] at MP3-8 on ST-Codecfake-OOD.Source: §VI · p. 6
  • Encoder interactions exceeded the prespecified ±5-point margin at six of twelve LA conditions and two of nine ST-Codecfake-OOD conditions; the ordering reverses at MP3-8 on ST.Source: §VI · p. 6
  • The registered omnibus test rejected across pairwise encoder interactions at in-support conditions on both corpora, with 11 and 7 contrasts beyond ±5 points.Source: §VI · p. 6
  • Equal weighting left LA’s material-condition set unchanged and raised ST-Codecfake-OOD’s count from two to three, with Opus-12 joining; retained interactions kept their signs.Source: §VI · p. 6

VII. RESULT 3: THE MATCHED COMPARISON WAS NOT ESTIMABLE, AND FIDELITY MEASURES RANK CONDITIONS DIFFERENTLY

The preregistered matched-fidelity comparison was not estimable, and waveform and perceptual measures ranked conditions differently. In the secondary support-region analysis, only H4 was supported under its registered rule, with a modest, noncausal association.

VII. RESULT 3: THE MATCHED COMPARISON WAS NOT ESTIMABLE, AND FIDELITY MEASURES RANK CONDITIONS DIFFERENTLY

  • Exactly one matched pair per corpus failed the gate; MP3 at 8 kbit/s ranked mid-grid on SI-SDR but last on PESQ-WB while costing the most attribution performance.The matched contrast was therefore not estimable under the preregistered rule on this grid.Source: §VII · p. 6

VIII. SECONDARY ANALYSIS: CONDITIONAL ASSOCIATION WITHIN THE SUPPORT REGION

  • H4 alone was supported: PESQ@M4 excluded zero and the R2 increment’s lower bound exceeded zero, with roughly one percentage point of additional explained variance.The result is noncausal evidence of non-redundancy within one support region, not an explanation of attribution failure.Source: §VIII · pp. 6–7, Table V · p. 7

VII. RESULT 3: THE MATCHED COMPARISON WAS NOT ESTIMABLE, AND FIDELITY MEASURES RANK CONDITIONS DIFFERENTLY

  • A 4 kHz bandwidth bottleneck residual was +14.8 [+6.2, +23.4] points, while 5 dB noise was −19.5 [−30.6, −8.5], opposing robust departures from fidelity-only prediction.The µ-law and 10 dB noise residuals were inconclusive; µ-law was outside codec support.Source: §VII · p. 6, Table IV · p. 7

VIII. SECONDARY ANALYSIS: CONDITIONAL ASSOCIATION WITHIN THE SUPPORT REGION

  • Sensitivity analyses left the H1–H4 truth table unchanged, and H2 remained inconclusive under both registered and equal weighting.H4 also survived a quadratic form; the class-signature-stratified audit returned PESQ@M4 −9.34 [−14.02, −4.67] and ΔR2 0.0110 [0.0043, 0.0193].Source: §VIII · p. 7
  • Across four corpus-by-encoder cells, the pooled neural coefficient was negative but nonuniform; only LA | W2V2-BERT had an interval excluding zero.The pooled estimate was numerically dominated by one cell, and collinearity constrained interpretation on LA.Source: §VIII · p. 7

IX. OPERATIONAL DIAGNOSTIC: THE REGISTERED OVERWRITE RULE WAS NOT CONFIRMED

The authors prospectively tested a frozen operational diagnostic for whether a rule could flag codec conditions associated with attribution failure; they explicitly distinguish it from a mechanism study.

IX. OPERATIONAL DIAGNOSTIC: THE REGISTERED OVERWRITE RULE WAS NOT CONFIRMED

  • The proposed neural-codec reading was converted into an executable decision procedure, verified byte-identical to development code, and tested prospectively as an operational diagnostic, not a mechanism study.The reading was that selected EnCodec conditions might replace rather than merely attenuate the generator signature.Source: A. The rule and what it was for · pp. 7–8
  • A cell received the representation-overwrite label only if all four interval conjuncts passed their prespecified ε = 0.05 margins.The four conjuncts test codec-direction selectivity, generator-signal margin loss, beyond-shrinkage relative to an isotropic-shrinkage null, and restoration after removing the codec mean shift.Source: A. The rule and what it was for · p. 8
  • The registered intersection–union decision required all four named LA EnCodec cells to receive the overwrite label and none of six DAC controls to do so.The control cells distinguished a universal neural-codec account from an EnCodec-specific account.Source: A. The rule and what it was for · p. 8

B. The registered outcome

The registered overwrite-rule decision failed its prespecified conjunctions. Descriptive labels varied by representation, and the prospective rule did not operate as a reliable flag on this grid.

B. The registered outcome

  • Both registered conjuncts failed: two of four named EnCodec cells missed the overwrite label, and three of six DAC controls received it.The registered decision was not prospectively confirmed.Source: B. The registered outcome · p. 8, Figure 5 · p. 9

C. Descriptive inspection of the frozen labels

  • Across ten registered LA cells, WavLM-Base+ received the overwrite label in 0 of 5 cells and W2V2-BERT 2.0 in 5 of 5.By codec family, the labels split 2 of 4 EnCodec and 3 of 6 DAC cells, so the grouping the rule targeted did not separate them.Source: C. Descriptive inspection of the… · p. 8, Figure 5 · p. 9

D. What the diagnostic shows

  • The fixed-in-advance rule failed as an operational flag on this grid, and representation-tracking labels do not establish that overwrite never occurs or that architecture causes it.The observed pattern did not support the deployment guidance implied by the rule.Source: D. What the diagnostic shows · p. 9

E. Why the rule looked promising on development data

A development-data rule looked promising, but its labels were explicitly qualified and its key predictions were contradicted prospectively. The discussion bounds the findings to tested conditions and recommends reporting transport performance across codec, fidelity, and representation dimensions.

E. Why the rule looked promising on development data

  • On development data, WavLM showed selective codec direction and large negative generator-margin changes for EnCodec, unlike MP3 and Opus.The permutation-null conjunct was waived and the shrinkage control was non-binding, so development labels used a modified rule.Source: E. Why the rule looked promising on… · pp. 9–10
  • WavLM at EnCodec-6k and EnCodec-3k changed from “representation overwrite” labels in development to “attenuation” prospectively.The authors state that development evidence was hypothesis-generating and carries no confirmatory weight.Source: E. Why the rule looked promising on… · p. 10

X. DISCUSSION

  • Codec-family comparisons without common support compare operating points rather than families, so the authors make no causal family claim.After adjustment, the neural-codec association pointed toward less degradation and was carried by one of four cells.Source: §X · p. 10
  • The study supports codec-transport performance loss within its tested scope, but does not establish why it occurs; its frozen mechanism-shaped rule was not confirmed.The primary decision covered two corpora, two deciding representations, and two head types, with descriptive evidence for a third representation.Source: §X · p. 10
  • A clean accuracy figure alone is insufficient because transport loss depends on codec condition, calibration, and threshold setting.All three encoders exceeded 0.85 clean Macro-F1, yet losses reached 53.5 and 70.3 points at one in-grid operating point.Source: §X · p. 10
  • A single fidelity scalar cannot order codec severity: matched-fidelity comparison was not estimable, and fidelity measures ranked conditions differently.The discussion also recommends evaluating multiple materially different representations because encoder interaction exceeded the ±5-point margin at six of twelve conditions on one corpus, with the more robust representation reversing between corpora.Source: §X · p. 10
  • Recovery, transcoding cascades, and attribution-score calibration under transport remain unaddressed; the tested grid is single-stage.Source: §X · p. 10

XI. LIMITATIONS

The authors limit their conclusions to closed-set tasks, two corpora, three representations, and a single-stage codec grid. They conclude that clean attribution accuracy does not guarantee robustness after transport, while several questions and implementation boundaries remain.

XI. LIMITATIONS

  • The matched-fidelity comparison was not estimable under the preregistered rule, leaving dependent registered clauses, including formal generality, non-executable.The study used 45 independent components on LA and 28 on ST-Codecfake-OOD; clip counts would overstate precision.Source: §XI · p. 10
  • Claims are bounded to two corpora, three representations, closed-set tasks, and a single-stage codec grid; no field-wide, open-set, cascade, or architecture-general result is claimed.The corpora have different label semantics, so attribution magnitudes are not compared across them.Source: §XI · p. 10

XII. CONCLUSION

  • For the tested scope, clean attribution accuracy does not guarantee codec-transport robustness; transport removed substantial performance on both corpora.The protocol fixed its analysis region from fidelity metadata before any attribution model was trained.Source: §XII · pp. 10–11
  • Transport losses vary across in-support conditions and representations, while waveform and perceptual measures rank conditions differently and matched-fidelity comparison was not estimable.The frozen development-based rule for flagging codec-induced failure was not confirmed prospectively.Source: §XII · p. 11

XIII. DATA AND CODE AVAILABILITY

  • The registered protocol, support masks, figure scripts, intervals, labels, and correction history are slated for release in a future revision.Source: §XIII · p. 11

XIV. DECLARATION ON THE USE OF GENERATIVE AI

  • Generative AI tools assisted only with language editing, phrasing, and manuscript preparation, according to the declaration.Source: §XIV · p. 11

APPENDIX A: CONDITION-LEVEL RESULTS

The appendix tables report condition-level Macro-F1 changes for the frozen codec grid on both corpora, with simultaneous bands restricted to in-support conditions.

APPENDIX A: CONDITION-LEVEL RESULTS

  • Table VI covers condition-level ΔMacro-F1 and simultaneous bands for ASVspoof 2019 LA; positive values denote attribution loss.Out-of-support conditions have point estimates only, with no decision weight.Source: Table VI · p. 12
  • Table VII reports condition-level ΔMacro-F1 with the same conventions for ST-Codecfake-OOD.Source: Table VII · p. 12
  • Simultaneous bands exist only for in-support conditions, where max-|t| calibration was computed; out-of-support estimates carry no intervals or decision weight.Source: Appendix A · pp. 11–12

APPENDIX B: ANALYSIS SPECIFICATIONS

The appendix defines a signed clip-level degradation response and specifies two analysis components: a fidelity-based model ladder and a frozen rule for diagnosing codec-related representation changes.

APPENDIX B: ANALYSIS SPECIFICATIONS

  • The signed degradation response compares clean and transformed clip correctness, with positive values for destroyed correct decisions and negative values for repaired incorrect decisions.The response is defined from the clean correctness indicator and condition-specific correctness indicator.Source: Appendix B · p. 12
  • The model ladder fits ordinary least squares and tests neural-family, rate, and PESQ estimands alongside incremental-fit terms.Models M3 and M4 add rate and PESQ terms, respectively, to the listed predictors.Source: Appendix B · p. 12
  • The frozen overwrite rule compares clean and transformed embeddings and uses four conjuncts, including codec-direction selectivity, beyond-shrinkage loss, and restoration.A cell is labelled representation overwrite only when all four conjuncts hold; otherwise it receives another specified label or remains unclassified.Source: Appendix B · pp. 12–13
  • The rule’s labels distinguish selective codec imprint with generator-margin loss, codec imprint without overwrite, attenuation, and unclassified cells.Restoration removes the codec mean shift rather than projecting onto the complement of its direction.Source: Appendix B · p. 13
Loading 2609.07981v1…