ASR-Protected Fusion and Reference-Anchored Policy Optimization for Joint ASR and Word-Level Speech Anti-Spoofing

Motivation

• Joint transcript generation with word-level spoof tags avoids mapping timeline-based predictions onto recognized words, where transcription or boundary errors can affect labels.The tagged transcript also carries authenticity information for downstream use.Source: §1 · p. 1

Approach

• APSGF restricts gated spoof-feature fusion to the orthogonal complement of a protected subspace formed from leading directions of bona fide ASR features.The selected ASR feature component is preserved while spoof-related information enters through the remaining directions.Source: §1 · p. 2
• RAPO compares one sampled response per input with a frozen post-SFT model’s greedy response, reinforcing responses with higher reward and suppressing those with lower reward.It uses joint transcription and spoof-label rewards without relying on within-group reward differences.Source: §1 · p. 2

Evaluation

• Across five datasets, the complete framework improves average ASR and word-level detection performance over the corresponding SFT baselines.The evaluation includes two-task and multi-task settings, with ablations under in-domain and out-of-domain conditions.Source: §1 · p. 2

2 Related work

• The related-work discussion is organized around spoof localization and ASR, feature fusion, and reward-guided sequence optimization.These three perspectives frame the paper’s treatment of word-tagged recognition, protected fusion, and reference-based policy feedback.Source: §2 · p. 2

2.1 Word-level anti-spoofing and ASR

• Prior work studies segment-level localization and tagged-ASR generation, which attaches spoof labels to recognized words without separate timestamp-to-word mapping at inference.Reported challenges include transcription difficulties and sensitivity to dataset-specific editing patterns.Source: §2.1 · p. 2

2.2 Feature Fusion and Subspace Constraints

• Audio-language models combine complementary encoders using approaches such as window-level Q-Formers, lightweight encoder selection, and prompt-aware mixtures; APSGF instead constrains spoof-feature fusion to ASR-orthogonal directions.Related work also applies gated fusion and orthogonal constraints in other speech and representation-learning settings.Source: §2.2 · pp. 2–3

2.3 Sequence Policy Optimization

• Sequence optimization methods differ in their feedback and baselines, while RAPO compares sampled responses with a fixed, low-stochasticity response from a frozen post-SFT model.The reference reward is computed once before optimization rather than changing with the current policy.Source: §2.3 · p. 3

3.1 Overall Framework Structure

• Speech is transcribed and spoofed words are enclosed by <TOF> and <EOF>, making transcription and word-level spoof detection one sequence-generation task.The framework uses supervised fine-tuning followed by RAPO.Source: §3.1 · pp. 3–4
• The Omni and spoof encoders provide complementary features, which are temporally aligned to Omni audio-token positions without word timestamps or transcript-word mapping.Spoof features are averaged over segments corresponding to Omni audio tokens.Source: §3.1 · p. 4

3.2 ASR-Protected Subspace Gated Fusion

• APSGF protects leading directions of bona fide Omni features and injects spoof information only into the orthogonal complement, leaving the protected-subspace projection unchanged.The protected subspace is constructed from bona fide VCTK training features; the complement contains directions available for spoof information.Source: §3.2 · pp. 4–5
• Separate gates control retained Omni features and injected spoof information in the complement, while initialization makes the fused representation approximately equal to the original Omni representation.The spoof MLP's last layer is initialized to zero and the Omni gate bias is set so its gate is approximately one.Source: §3.2 · p. 5

3.3 Reference-Anchored Policy Optimization

• RAPO samples one completion per input and compares its reward with a fixed reward from the post-cross-entropy model's greedy completion.The reward evaluates transcription and word-level spoof-label errors.Source: §3.3 · p. 5
• Positive advantages reinforce sampled completions and negative advantages suppress them, with length-normalized sequence log-probability optimization and KL regularization constraining updates.The reference reward remains fixed throughout training.Source: §3.3 · p. 5

4.1 Experimental Setup

• Training uses four datasets, while LlamaPartialSpoof—with more than 30,000 utterances and edits from six TTS systems—is reserved for out-of-domain evaluation.The datasets cover bona fide, partially manipulated, and fully synthetic speech; the framework is evaluated in two-task and multi-task settings.Source: §4.1 · pp. 5–6

4.2 Experimental Results

• Ours_M achieves the best average utterance-level accuracy, 0.952, including 0.805 on out-of-domain LlamaPartialSpoof.Ours_T averages 0.824 and reaches 0.454 on LlamaPartialSpoof.Source: Table 2 · p. 7, §4.2 · p. 7
• Average out-of-domain WordF1 decreases from 0.838 for the two-task model to 0.759 for the multi-task model.The authors suggest scope-prediction errors may affect subsequent word tags, but note that this requires further analysis.Source: §4.2 · p. 7, Table 1 · p. 7
• Combining APSGF and RAPO achieves the lowest average WER, 0.054, while maintaining WordF1 of 0.901.The ablation results attribute complementary gains to the fusion and optimization modules.Source: Table 3 · p. 8, §4.2 · p. 8
• Compared with GRPO, RAPO reduces average WER from 0.068 to 0.054 and raises WordF1 from 0.872 to 0.901.The reported comparison says GRPO brings no clear overall improvement.Source: Table 4 · p. 8, §4.2 · p. 8

5 Conclusion and Discussion

• Across five datasets, the framework improves average ASR and word-level spoof detection, while LlamaPartialSpoof results show out-of-domain word-level detection remains challenging.The conclusion describes complementary gains from the auxiliary encoder, APSGF, and RAPO.Source: §5 · p. 8
• APSGF adds spoof cues while preserving ASR representations, and RAPO anchors policy optimization to the post-SFT model.The conclusion characterizes their gains as complementary across spoof representation, localization, and transcription.Source: §5 · p. 8
• The authors suggest extending the framework to acoustic-semantic tasks and applying RAPO where a model changes little after supervised fine-tuning.They frame reference anchoring as a possible way to add acoustic objectives while preserving existing audio-language model capabilities.Source: §5 · p. 8

Limitations

• The framework is evaluated only for word-level speech anti-spoofing, not other acoustic-semantic tasks.Its applicability beyond this task is not validated.Source: Limitations · p. 9
• Experiments use a single audio-language model, so generality across architectures remains unclear.Source: Limitations · p. 9
• The tagged ASR formulation mainly identifies spoofed words and cannot detect deletion-based edits.Source: Limitations · p. 9
• Hyperparameter tuning is limited, and further optimization may yield additional gains.Source: Limitations · p. 9

6 Ethical Considerations

• The datasets used in the work are publicly available and used in accordance with their licenses and terms.Source: §6 · p. 9
• LLMs were used solely to polish the manuscript for clarity and readability.Source: §6 · p. 9
• The authors state that LLMs were not used for ideation, methodology, experimental design, data analysis, or result interpretation.Source: §6 · p. 9

A Dataset Details

• Only English data with available transcriptions are used, so some datasets contribute only a subset.Source: A Dataset Details · p. 11
• Training combines VCTK, PartialEdit, AV-Deepfake1M, and SCDF, covering bona fide, partially spoofed, and fully spoofed utterances.PartialEdit uses E1 and E2; AV-Deepfake1M includes only speech deepfakes, and SCDF is randomly split four-fifths for training and one-fifth for testing.Source: A Dataset Details · p. 11
• Held-out splits provide evaluation, while the R01TTS.0.b subset of LlamaPartialSpoof is the out-of-domain test set.Source: A Dataset Details · p. 11

B.1 Two-Task Setting

• The two-task prompt requests transcription and marks only spoofed words with <TOF> and <EOF>, with plain transcription for no-spoof audio.The examples contrast none, partial, and all-spoof cases, including wrapping only the altered word in the partial case.Source: §B.1 · p. 11

B.2 Multi-Task Setting

• The multi-task prompt adds a spoof-level classification before transcription and word-level spoof tagging.Its examples provide none, partial, and all labels for utterance-level spoof scope.Source: §B.2 · p. 11

C Auxiliary Spoof Encoder Gate Fusion Details

• The baseline maps spoof features to the Omni feature dimension with an MLP before fusion.Source: C Auxiliary Spoof Encoder Gate Fusion… · p. 12
• Two independent gates are predicted from concatenated Omni and spoof features.One gate controls Omni-feature contribution and the other controls projected spoof-feature contribution.Source: C Auxiliary Spoof Encoder Gate Fusion… · p. 12
• At each temporal position, the gated Omni and spoof features are added, then fused features are concatenated across time.Source: Figure 3 · p. 11, C Auxiliary Spoof Encoder Gate Fusion… · p. 12
• The baseline initializes the MLP’s final layer to zero and sets the Omni-gate bias to initialize from original Omni representations.The initial fused representation is approximately the Omni representation.Source: C Auxiliary Spoof Encoder Gate Fusion… · p. 12
• The fusion baseline removes APSGF’s ASR-protected subspace constraint while retaining dual-gate fusion across the full Omni feature space.This comparison isolates the effect of subspace protection on joint ASR and word-level spoof detection.Source: C Auxiliary Spoof Encoder Gate Fusion… · p. 12

• GRPO starts from the epoch-5 SFT checkpoint with a fresh AdamW optimizer and trains for one epoch using a 5 × 10−6 learning rate, cosine decay, and 0.03 warmup.Source: D GRPO Implementation Details · p. 12
• Table 6 reports total instantiated parameters and trainable DoRA weights with their associated modules for the ablation models.Source: Table 6 · p. 12
• GRPO samples eight responses per prompt at temperature 0.90, top-p 0.95, and top-k 50, with a maximum generation length of 1024 tokens.Source: D GRPO Implementation Details · p. 12
• GRPO uses RAPO’s reward function and KL settings, with a KL coefficient of 0.04.Source: D GRPO Implementation Details · p. 12

• Total parameter counts include frozen Qwen2.5-Omni Thinker and, where applicable, the frozen Wav2Vec2-XLS-R-300M encoder.Source: E Parameter Counts of Ablation Models · p. 12
• Trainable parameter counts comprise DoRA weights and their associated modules.Source: E Parameter Counts of Ablation Models · p. 12
• RAPO adds no model parameters, whereas APSGF increases the count with an auxiliary feature extractor and additional gating networks.Source: E Parameter Counts of Ablation Models · p. 12