Does a prosody-trained representation help beyond trainable fusion? A parameter-matched study with frozen HuBERT

1. INTRODUCTION

• Spontaneous speech reduction varies with predictability and prosodic prominence, motivating tests of whether acoustic-prosodic cues help recognize ambiguous speech.Prominence-associated cues include F0, loudness, duration, and spectral balance.Source: §1 · p. 1
• SSL models encode prosodically relevant information, but recoverability alone does not establish that downstream ASR exploits it during recognition.The paper frames this distinction as a question about effective use, not merely whether cues are present in representations.Source: §1 · p. 1
• A frozen SSL baseline versus an auxiliary-conditioned model changes both the supplied information and the trainable mechanism, preventing unique attribution of any recognition gain to the auxiliary information.With the backbone frozen, the added mechanism provides another pathway for transforming fixed SSL representations.Source: §1 · p. 1
• The study compares Learned, which receives a prosody-trained representation, with an architecturally and parameter-matched Null condition that receives zeros.Null–Baseline separately assesses the effect of adding trainable fusion modules.Source: §1 · p. 1
• Across three corpora, trainable fusion consistently improves recognition, while the learned prosody representation adds little WER improvement.Inference-time interventions test whether the trained Learned model uses its auxiliary representation.Source: §1 · p. 1

2. RELATED WORK

• Prosodic information is recoverable from SSL representations, but that does not show ASR effectively uses it during recognition.Prior work reports prosody reconstruction, future-prosody prediction, and strong-versus-weak boundary discrimination.Source: §2 · p. 1
• Prior ASR work jointly trained pitch-accent detection and recognition, but its comparison left open whether gains reflected prosodic supervision or the broader auxiliary objective.This study instead holds fusion fixed while varying whether the input is informative.Source: §2 · p. 2
• Probing research shows that trainable heads can affect evaluation and model rankings, while this study controls that pathway and separately tests representation use at inference.Related ASR systems also inject auxiliary cues through trainable pathways.Source: §2 · p. 2

3.1. Overview

• A frozen HuBERT backbone with post-hoc layerwise fusion is used to test the separate contributions of the prosody representation and trainable fusion mechanism.Modified layer states are used for downstream aggregation rather than passed into the next HuBERT layer.Source: §3.1 · p. 2

3.2. Data

• Phase 2 evaluates on Buckeye, Switchboard, and AMI IHM, while the Phase 1 encoder is trained and selected using a disjoint CASPER split.The three ASR corpora span close-microphone spontaneous speech, telephone conversations, and headset-recorded meetings.Source: §3.2 · p. 2

3.3. Phase 1: Prosody-trained encoder

• The Phase 1 encoder produces 64-dimensional, 20 ms-frame representations supervised by log F0, voicing, Delta log F0, log energy, and spectral tilt.Undefined targets are masked rather than imputed, and the objective combines mean-squared error for continuous targets with binary cross-entropy for voicing.Source: §3.3 · pp. 2–3

3.4. Phase 2: Post-hoc HuBERT fusion

• The frozen Phase 1 representation is aligned to HuBERT frame lengths and conditions FiLM-style gated residual fusion modules on transformer-layer states.The pre-transformer state bypasses fusion; a weighted sum of all thirteen states feeds a projection, BiLSTM, and CTC output layer.Source: §3.4 · p. 3

3.5. Experimental conditions and training

• Learned and Null have identical fusion architectures and 12.16M trainable parameters, while Baseline omits fusion and has 10.93M.Only the downstream components are optimized; HuBERT and the Phase 1 encoder remain frozen.Source: §3.5 · p. 3

3.6. Statistical analysis

• The primary Learned–Null comparison tests the auxiliary representation against zero input with fusion held fixed; Null–Baseline is secondary.Paired test-set predictions are analyzed with a hierarchical Poisson bootstrap, 100,000 draws, and separate Holm corrections for the two sets of corpus-level tests.Source: §3.6 · p. 3

3.7. Inference-time diagnostics

• Inference diagnostics evaluate each trained Learned checkpoint with its original, zeroed, time-shuffled, or other-utterance representation without retraining.ZERO is an out-of-distribution sanity check because Learned was not trained with an all-zero auxiliary sequence; intervention contrasts are exploratory.Source: §3.7 · p. 3
• Fusion residual magnitudes are quantified relative to frozen HuBERT states and reported descriptively, without post-hoc significance tests across layers.The residual-magnitude equation defines the layerwise quantity being compared.Source: §3.7 · p. 3

4.1. Recognition performance

• Learned–Null WER differences were +0.069 on Buckeye, −0.090 on Switchboard, and +0.004 on AMI IHM; none was significant, while Null beat Baseline by 0.71–1.45 points.The confidence intervals are compatible with at most a few tenths of a percentage point of improvement from the learned representation under this architecture.Source: Table 1 · p. 4, §4.1 · p. 4

4.2. Functional use of the auxiliary representation

• 0.42–1.06-point utterance substitutions significantly increased WER across corpora; temporal shuffling raised it 0.17–0.35 points, while zeroing caused 4.85–16.93-point out-of-distribution degradations.Shuffling was significant for Switchboard and AMI IHM, but not Buckeye; zeroing is a sanity check, not direct evidence of incremental utility.Source: Table 2 · p. 4, §4.2 · p. 4

4.3. Fusion behavior

• Null modified frozen HuBERT states more than Learned on every corpus, while Learned showed its strongest modifications around layers 9–11.Mean relative residual magnitudes for Null versus Learned were .351 versus .249 on Buckeye, .825 versus .526 on Switchboard, and 1.091 versus .672 on AMI IHM.Source: Figure 2 · p. 4, §4.3 · p. 4

• The trainable fusion mechanism improves the frozen HuBERT recognizer, whereas the prosody-trained representation yields little additional WER reduction.Under this architecture, most observed gain is associated with the fusion pathway rather than the auxiliary representation.Source: §5 · p. 4
• One possible explanation is that frozen HuBERT states already encode information correlated with the supervised acoustic-prosodic cues, leaving little incremental information for the auxiliary representation.Learned nevertheless depends on its auxiliary input: utterance substitution degrades recognition even though Null achieves comparable WER without an utterance-varying signal.Source: §5 · p. 4
• The conclusions are limited to one SSL backbone, supervision that omits duration, a CTC recognizer, and a frozen post-hoc design.Full fine-tuning, propagating fused states through HuBERT, or localized error analysis may yield different effects.Source: §5 · p. 4

• The study was conducted retrospectively using previously collected human speech data from CASPER, Buckeye, Switchboard, and AMI.Source: §6 · p. 5
• The authors report that no funding was received for the study.Source: §6 · p. 5
• The authors disclose no relevant financial or nonfinancial interests.Source: §6 · p. 5