Qwen-Audio-Jev: A Unified Interface for Audio Decisions

1 Introduction

• Autoregressive generation adds computational overhead and latency for bounded outputs, while generated answers may also need parsing and validation in high-throughput applications.The motivation is a direct readout that retains natural-language task specification but produces answers in an explicitly defined space.Source: §1 · pp. 1–2
• Qwen-Audio-Jev augments Qwen-Audio-3.1-ASR-Flash-Next with a dedicated head that directly produces bounded decisions from audio, a natural-language question, and an answer specification.It retains the backbone’s audio encoder, adaptor, and MoE language decoder, reading out decisions at the decoder’s pre-answer hidden state.Source: §1 · p. 2
• The head unifies selection, binary judgment, and scoring as CHOICE distributions over supplied alternatives, NOUL probabilities for binary propositions, and SCORE distributions over ordered levels.The interface is inspired by Jev’s typed decision primitives (Almeida, 2026; TypeSafe AI, 2026).Source: §1 · p. 2
• A common supervision representation lets examples from heterogeneous datasets train only the tasks supported by their available labels, without requiring every recording to carry every annotation.The formulation preserves distinctions among categorical labels, binary relations, and ordered ratings.Source: §1 · p. 2
• 78.70% accuracy on MMAU-mini, 84.06% on MMSU, and 6.50% equal error rate on VoxCeleb1-O accompany a 0.1062 normalized MAE on NISQA overall quality.Evaluation covers 18 labeled configurations; the model also achieves an approximately 20× speedup over the original backbone in the evaluated inference setting.Source: §1 · p. 2

2 System Architecture

• The system architecture review covers the Qwen-Audio backbone and describes how its input-conditioned representations are converted into CHOICE, NOUL, and SCORE outputs.The section first reviews Qwen-Audio-3.1-ASR-Flash-Next, then introduces the Jev decision head.Source: §2 · p. 2

2.1 Review of Qwen-Audio-3.1-ASR-Flash-Next

• The backbone encodes audio, adapts its representation to the decoder, and combines it with textual instructions and context to produce task-dependent responses.Qwen-Audio-Jev retains this pathway, but obtains typed answers from a decision head rather than an autoregressively decoded answer sequence.Source: §2.1 · pp. 2–3

2.2 Jev decision-head implementation

• The decision head reads the final question-and-specification prefix state before answer tokens, then uses three affine projections selected by the requested primitive.Choice and Score normalize only valid candidate or level positions; unused positions are excluded, and their meanings come from the answer specification.Source: §2.2 · p. 3, Figure 1 · p. 4

2.3 Typed outputs: Choice, Noul, and Score

• CHOICE returns probabilities over supplied alternatives, NOUL a proposition’s truth probability, and SCORE an expected value over ordered rating levels.Score’s native output is a probability-weighted level index; it can be mapped to application units, with reference ratings using the same mapping during evaluation.Source: §2.3 · pp. 4–5

2.4 Inference procedure

• Inference encodes the audio, processes the question and answer specification, and applies the requested readout to the pre-answer state without an autoregressive answer-decoding loop.Questions in a multi-question request are evaluated separately, requiring multiple question-conditioned forward passes.Source: §2.4 · p. 5

• Training uses supervised fine-tuning (SFT) for the three primitives, followed by reinforcement learning for calibrated decisions (RLCD) from an SFT checkpoint.SFT jointly optimizes the audio encoder, adaptor, language-model backbone, and Jev head; RLCD optimizes a calibration-oriented reward through sampled-action policy gradients.Source: §3 · p. 5

3.1 Supervised Fine-Tuning

• SFT jointly optimizes the audio encoder, adaptor, language-model backbone, and Jev decision head, activating only the objective for the requested primitive.The other readouts receive no task-specific supervision.Source: §3.1 · p. 5

3.1.1 Training formulation and supervision

• Training examples use a unified audio–question–criteria–target format, and recordings need labels only for the primitives with observed, reliable annotations.Missing annotations contribute no loss; Choice, Noul, and Score targets use categorical labels, binary judgments, and explicitly ordinal audio-grounded annotations, respectively.Source: §3.1.1 · pp. 5–6

3.1.2 Primitive-specific objectives

• Choice uses categorical cross-entropy over valid candidates, with hard labels yielding one-hot targets and soft targets supported when annotation distributions are available.The objective covers nominal categories and multiple-choice audio questions.Source: §3.1.2 · pp. 6–7
• Noul uses binary cross-entropy for binary propositions and also supports fractional annotations.Speaker verification and yes/no audio judgments use this objective.Source: §3.1.2 · p. 7
• Score combines level-distribution, cumulative-distribution, and expected-rating losses, with the latter two weighted by 0.25 and 0.10 in all experiments.The expected rating is constrained on the native [0, Ki −1] scale.Source: §3.1.2 · p. 7

3.1.3 Overall objective and task sampling

• Each loss-reduction call processes one observed question, so the training-stream sampling distribution controls the relative exposure of the three primitives.The SFT objective also includes the backbone load-balancing loss and router z-loss; no next-token or multi-token prediction objective is included.Source: §3.1.3 · p. 7

3.2 Reinforcement Learning for Calibrated Decisions

• RLCD initializes from SFT and formulates each typed decision as a one-step contextual-bandit problem optimized with sampled-action REINFORCE.Its calibration-oriented reward draws on quadratic/Brier and ranked-probability scoring rules.Source: §3.2 · p. 8

3.2.1 Decision policies and exploration

• Choice and Score use categorical policies over candidates or ordered levels, while Noul uses a two-class false/true policy conditioned on audio and question.The decision state is extracted before the answer, and the observed outcome evaluates sampled decisions.Source: §3.2.1 · p. 8

3.2.2 Calibration-oriented group reward

• The reward combines agreement with the observed outcome and a pairwise term characterizing the concentration of the sampled decision distribution.An ordinal term is omitted for Choice and Noul.Source: §3.2.2 · p. 8

3.2.3 Policy-gradient optimization

• The policy-gradient update subtracts a leave-one-out baseline independent of the sampled action whose log-probability is differentiated.Gradients flow through sampled-action log-probabilities, while actions, rewards, returns, baselines, and importance weights are treated as constants; the expected surrogate gradient equals the gradient of Bi + λiDi.Source: §3.2.3 · p. 9

4.1 Evaluation setup

• The evaluation covers 18 labeled configurations: eight Choice tasks, four Noul tasks, and six Score tasks.Choice covers classification and audio question answering; Noul includes speaker verification and other binary judgments; Score covers speech-quality assessment.Source: §4.1 · pp. 9–10

4.2.1 Choice: categorical decisions

• Speaker-gender and human non-speech recognition both exceed 90% accuracy, while conversational emotion recognition is the weakest Choice configuration.The gap is not simply speech versus non-speech: speaker attributes and vocal-event identity are handled more accurately than conversational affect.Source: §4.2.1 · p. 10, Table 2 · p. 10
• MMSU performs best among the three audio-question-answering benchmarks, followed by MMAU and MMAR.MMAR’s lower accuracy makes complex question-conditioned answer selection a separate improvement target from emotion recognition.Source: §4.2.1 · p. 10

4.2.2 Noul: binary judgments

• PodSarc and VoxCeleb1-O both achieve AUC above 0.98 and FPR and FNR below 8% at threshold 0.5; PodSarc also leads on EER and Brier score.Their performance is consistent across probability-based and binary-decision measures.Source: §4.2.2 · pp. 10–11, Table 3 · p. 10
• Clotho-AQA and In-the-Wild have nearly identical AUCs (0.8787 versus 0.8763) and EERs around 20%, but In-the-Wild has 57.29% FNR versus 24.49%.At threshold 0.5, In-the-Wild’s errors are dominated by missed positives rather than false alarms.Source: Table 3 · p. 10, §4.2.2 · p. 11

4.2.3 Score: subjective quality assessment

• NISQA overall quality and coloration combine comparatively low MOS errors with correlations close to 0.8; loudness has MAE close to overall quality but lower Pearson correlation.Similar average errors can accompany different levels of linear association with human ratings.Source: §4.2.3 · p. 11, Table 4 · p. 11
• VoiceMOS has Pearson and Spearman correlations of 0.723 and 0.754, but its MOS MAE of 0.813 exceeds every NISQA dimension.Noise has the largest NISQA MAE at 0.584; the stated priorities are reducing MOS errors for VoiceMOS and noise quality, and improving loudness rating consistency.Source: §4.2.3 · p. 11, Table 4 · p. 11

5 Conclusion

• Qwen-Audio-Jev reports 84.06% accuracy on MMSU, 6.50% EER on VoxCeleb1-O, and NISQA overall MAE of 0.4246 MOS points with Pearson correlation of 0.8210.The conclusion identifies emotion recognition, complex audio question answering, and quality-scoring accuracy as further improvement areas.Source: §5 · p. 11
• The model adds a decision head for CHOICE, NOUL, and SCORE, producing question-conditioned outputs without autoregressive answer generation.Its training uses primitive-specific objectives and heterogeneous supervised fine-tuning, followed by an SFT-initialized RLCD stage with exploratory sampling, label-driven group rewards, and reward-weighted policy updates.Source: §5 · p. 11
• Streaming decisions are planned as a future extension, with chunkwise predictions, confirmation deferred when an audio prefix is insufficient, and evaluation focused on decision quality, stability, and end-to-end latency.Source: §5 · p. 11