Text Scores Do Not Establish Performance on Lexically Non-Diagnostic Speech Tasks: A Qwen2-Audio Quantization Case Study

1 INTRODUCTION

• Two recordings can share the same words but carry different emotion labels, so a good transcript does not establish emotion-task performance.The transcript cannot determine the label, though this task property does not identify which acoustic cues the model uses.Source: §1 · p. 1
• The study asks what evidence is needed before text-output scores describe quantized-model performance on speech tasks beyond the transcript.It reports translation scores, transcript-nondiagnostic emotion recognition, an FP16 baseline, and matched-bit-budget controls.Source: §1 · p. 1
• The evaluation protocol separates text output from performance on tasks whose labels cannot be recovered from transcripts, with an FP16 baseline, matched budgets, and independent-unit uncertainty.Source: §1 · p. 2
• The same fixed 6- and 7-bit Qwen2-Audio allocations are evaluated on FLEURS and RAVDESS against FP16 and matched-budget controls.Source: §1 · p. 2
• −3.71% is the 6-bit RAVDESS accuracy difference from FP16, with a speaker interval excluding zero and exact sign-flip p = 0.0148; the 7-bit interval includes zero.Both selected allocations have unresolved FLEURS differences from FP16, and neither leads both matched controls.Source: §1 · p. 2

2 RELATED WORK

• Prior PTQ approaches for large language models include LLM.int8(), GPTQ, SmoothQuant, and activation-aware weight quantization (Frantar et al., 2022; Lin et al., 2023).Speech-specific work covers mixed-precision PTQ, speech translation compression, codec compression, and speech language models (Moslem, 2025).Source: §2 · p. 2
• Audio-language evaluation studies direct audio use, temporal grounding, and token-level readout, while LISTEN intervenes on lexical and acoustic emotion cues across six models (Zhang et al., 2026; Dai et al., 2026; Kulkarni et al., 2026a; Ma et al., 2026; Kulkarni et al., 2026b; Chen et al., 2026).Here, balanced sentence frequencies do not isolate acoustic cues; the study tests fixed quantized allocations against FP16 and same-budget controls without identifying cue use or proposing a new benchmark.Source: §2 · p. 3

3 EVALUATION PROTOCOL

• The study evaluates translation on 508 English-to-German FLEURS utterances, sharing examples only for FP16 and both selected allocations, and emotion recognition on 512 RAVDESS clips from 16 speakers, with all seven configurations sharing clips.RAVDESS has four clips per actor–emotion pair, and each spoken sentence occurs 32 times per emotion.Source: §3 · p. 3, §3.2 · p. 5

3.1 TWO DIFFERENT QUESTIONS

• BLEU and chrF measure generated English-to-German text, not an emotion label that the transcript cannot determine.The metrics use one reference translation per utterance and specified SacreBLEU settings.Source: §3.1 · p. 3, §3.2 · p. 4
• Both spoken sentences occur equally often with each of eight RAVDESS labels, making sentence identity alone a 12.5% accuracy rule, not a measured transcript-only model baseline.Above-chance accuracy does not identify acoustic cues as the source; separating those cues requires an intervention.Source: §3.1 · p. 3

3.2 QWEN2-AUDIO CASE STUDY

• For each budget, the selected plan maximizes development chrF among seven candidates and is fixed before test evaluation; modules start at 4 bits, with the output head at 8 bits.The plans were chosen on a separate 387-example translation development split, not RAVDESS.Source: §3.2 · p. 4
• Front-block and stratified-uniform controls change which modules receive 8-bit precision while keeping average budgets effectively fixed.Matched average precision constrains resource allocation but does not match module quantization error or guarantee equal activation behavior.Source: §3.2 · p. 4, §3.1 · p. 4
• RAVDESS scoring accepts exactly one normalized emotion label, while greedy decoding holds the decoding rule fixed.The outcome is categorical agreement with acted labels, not an explanation, confidence score, or listener judgment.Source: §3.2 · p. 5
• FLEURS intervals resample paired utterances, while RAVDESS intervals resample speakers and the sign-flip test enumerates paired speaker differences.Intervals including zero leave a named comparison unresolved rather than establishing equal performance; tests are not adjusted for multiple comparisons.Source: §3.2 · p. 5

4.1 TRANSLATION-SCORE DIFFERENCES FROM FP16 REMAIN UNRESOLVED

• Every FLEURS BLEU and chrF interval for the 6- and 7-bit allocations includes zero, leaving measured differences unresolved on this fixed test list.The study specifies no acceptable-loss margins for interpreting these differences as preservation.Source: Table 2 · p. 6, §4.1 · p. 6

4.2 RAVDESS IDENTIFIES A 6-BIT FP16 DEFICIT

• On RAVDESS, the 6-bit allocation differed from FP16 by −3.71% (95% speaker interval [−6.05, −1.37]%; p = 0.0148), while the 7-bit difference was −1.17% ([−2.73, +0.20]%; p = 0.2656).The speaker interval excludes zero for 6 bits but includes zero for 7 bits.Source: §4.2 · p. 6, Figure 2 · p. 7
• Neither selected allocation has the highest observed RAVDESS accuracy at its bit budget; only the 7-bit selected-minus-front-block interval excludes zero.The other reported matched-control intervals include zero.Source: Figure 2 · p. 7, §4.2 · p. 7

4.3 SPEAKER-LEVEL DECOMPOSITION

• Across actors, 6-bit differences were negative for 12 of 16, while 7-bit differences were zero for nine; exact sign-flip tests gave p = 0.0148 and p = 0.2656, respectively.The sign-flip analysis uses one paired mean difference per actor and relies on sign-exchangeability under its sharp null.Source: §4.3 · pp. 7–8, Table 3 · p. 8

5 DISCUSSION

• Both FLEURS BLEU and chrF intervals include zero at both budgets; on RAVDESS, 6-bit accuracy is lower than FP16 while the 7-bit difference is unresolved.The selected 6-bit allocation is lower than FP16 under both uncertainty analyses.Source: §5 · p. 8
• The cross-task comparison does not isolate a shared acoustic representation or establish a causal translation–emotion trade-off.The tasks have different populations and outputs.Source: §5 · p. 8
• Each allocation changes reconstructed weights in multiple modules, so matched average precision does not identify a module-level cause.Localizing a cause would require interventions on specified modules while holding the remaining allocation fixed.Source: §5 · p. 8
• Neither selected placement shows an observed emotion-accuracy advantage over both same-budget controls.Selected-plan point estimates are lower at both budgets, with a resolved adverse 7-bit front-block comparison and remaining control pairs unresolved.Source: §5 · pp. 8–9
• The RAVDESS endpoint is lexically non-diagnostic but does not prove that predictions use emotion-related acoustic cues.Speaker identity, recording conditions, or other non-lexical information may also contribute; these boundaries do not describe deployment behavior.Source: §5 · p. 9
• For intended emotion or other transcript-absent targets, evaluate that target against a full-precision baseline and same-budget controls.A translation score cannot replace that measurement, and this study does not establish a general trade-off or predict other quantized speech models.Source: §5 · p. 9
• Transcript-only conditions and word-preserving audio perturbations are proposed to test whether information beyond words helps and which cues the model uses.Evaluation across checkpoints and naturalistic emotion data would test the scope of the accuracy result.Source: §5 · p. 9
• Joint translation-and-emotion allocation requires explicit objectives because the task outcomes use different units.Combining them requires a stated weighting or acceptable loss; the study specifies neither a preferred weighting nor the best joint-objective allocation.Source: §5 · p. 9

6 LIMITATIONS

• The experiments use one Qwen2-Audio checkpoint and one acted North American English emotion set, limiting the evaluated scope.Results may differ for spontaneous emotion, long-form speech, multilingual prosody, or other transcript-independent targets.Source: §6 · p. 9
• FLEURS intervals describe test-utterance variation for one generation run, while RAVDESS inference uses 16 speaker clusters and assumes sign-exchangeability under the null.Sentence balance does not distinguish emotion cues from speaker or recording cues, or establish causal audio use.Source: §6 · p. 9
• Exact-match accuracy measures agreement with acted dataset labels under the specified prompt, not perceived emotion quality or listener usefulness.No human listening or human emotion judgment was performed.Source: §6 · p. 9

7 CONCLUSION

• Translation-score differences from FP16 remain unresolved at 6 and 7 bits, while RAVDESS accuracy is lower at 6 bits and unresolved at 7 bits.Neither selected allocation has higher observed accuracy than both same-budget controls.Source: §7 · p. 9
• Separate evaluation of text quality and the intended speech task should use a full-precision reference, matched budgets, and uncertainty estimates accounting for speaker dependence.Text-output scores alone do not establish performance on targets that cannot be recovered from the transcript.Source: §7 · p. 9
• Future work includes other checkpoints and naturalistic emotion data, cue-use tests, and human evaluation of perceived emotion understanding.Proposed cue-use tests include transcript-only conditions and controlled audio perturbations.Source: §7 · p. 9

AI USE STATEMENT

• AI tools helped find related work and draft and edit text.The statement reports assistance with manuscript work.Source: AI USE STATEMENT · p. 10
• AI tools also helped review the evaluation and its interpretation and revise figure code and layout.Source: AI USE STATEMENT · p. 10
• The authors checked citations, numerical results, and the rendered paper against cited sources and stored experiment outputs, and take responsibility for the final manuscript and artifacts.Source: AI USE STATEMENT · p. 10

• Section 3 describes the test populations, precision allocations, scoring rules, and uncertainty analyses.Source: REPRODUCIBILITY STATEMENT · p. 10
• The linked reproducibility repository provides allocation definitions, scoring code, and metric configurations.Source: REPRODUCIBILITY STATEMENT · p. 10

• The selected allocations have lower observed accuracy than both controls; intervals exclude zero for 6-bit versus FP16 and 7-bit versus front-block.Source: Table 4 · p. 12, A RAVDESS ALLOCATION COMPARISONS · p. 12
• Front-block has the highest observed accuracy at 7 bits, while stratified-uniform has the highest at 6 bits; unresolved intervals do not establish either control as better.The unadjusted tests do not provide a familywise guarantee.Source: A RAVDESS ALLOCATION COMPARISONS · p. 12