PaperlayerView exact revision on arXiv

Source-linked AI summary

AudioJev: Direct Audio Decisions with Order-Calibrated Probabilities

Sihan Lv, Zhen Li, Zhiqi Cao, Jinshan Zhang, Ying Li, Meng Xi, Jianwei Yin

arXiv:2610.01293v1cs.SD

TL;DR

Audio decisions can depend on evidence that transcripts do not preserve, while serialized answer order can affect probability estimates. AudioJev trains a shared waveform-to-candidate model with supervised, identity-aligned reordered views; it reports results on complete MMAU and MMAR benchmarks.

  • Problem

    Transcribe-then-classify can restrict decisions to lexical evidence, and answer ordering can change language-model probabilities for an unchanged audio question.

    Source: §1 · p. 1

  • Method

    AudioJev maps audio, a question, and supplied alternatives to candidate probabilities, training reordered views with supervision and identity-aligned symmetric KL.

    Source: §1 · pp. 1–2

  • Results

    AudioJev reports 68.88% on complete MMAU and 55.33% on complete MMAR.

    Source: §1 · p. 2

  • Takeaways & Limitations

    One shared model handles semantic, acoustic, musical, and conversational decisions through a candidate-conditioned interface.

    Source: §1 · p. 1

    The interface targets closed-set decisions with caller-supplied admissible answers; conversational evaluation is offline at proposed pauses, not within an agent’s control loop.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Audio decisions often depend on evidence that a transcript does not preserve, while their probability estimates can depend on how answer options are ordered. AudioJev maps a waveform, question and supplied alternatives directly to a candidate distribution through one shared full-parameter model. We define order calibration as preserving an answer's probability under meaning-preserving option permutations. Random-derangement SKL training pairs each question with a reordered view in which every alternative changes position, supervises both answers, and aligns the two distributions before applying a symmetric KL penalty. Inference retains a single candidate-scoring forward, with no calibration head or order ensemble. Across three training seeds, AudioJev reaches 68.88%/55.33% mean accuracy on complete MMAU/MMAR and reduces random-order SKL by 43.8%/60.5% relative to the single-view removal ablation. The same model handles intent, environmental sound, note properties, speech activity and conversational transitions. Paired removal ablations and multi-order evaluation measure predictive accuracy and probability stability together, establishing a direct audio interface whose calibration objective acts on candidate meaning rather than presentation position. Inference code and model weights are available at https://github.com/SihanLv/AudioJev-Inference and https://huggingface.co/shlv/AudioJev.

1 Introduction

AudioJev addresses transcript-limited audio decisions and answer-order-sensitive probabilities with a shared candidate-conditioned model. Its paired training objective aligns probabilities across reordered answer lists, and the evaluation reports complete-benchmark results alongside order-sensitivity diagnostics.

1 Introduction

  • Transcribe-then-classify systems can lose environmental, musical, or conversational evidence, while serialized answer positions can affect language-model probabilities (Radford et al., 2023; Deshmukh et al., 2023; Xu et al., 2025; Pezeshkpour and Hruschka, 2024; Zheng et al., 2024).A purely presentational edit can change probabilities for an unchanged audio question.Source: §1 · p. 1
  • AudioJev maps waveforms, questions, and caller-supplied answers to probabilities across semantic, acoustic, musical, and conversational tasks using shared parameters.Candidate-only supervision keeps training and inference on the same answer support.Source: §1 · pp. 1–2
  • Random-derangement SKL supervises both answer-list views and aligns their distributions by answer identity, making presentation consistency part of learning.Every alternative changes position in the paired view, while inference still uses one model forward.Source: §1 · pp. 1–2
  • 68.88% on complete MMAU and 55.33% on complete MMAR are the final model’s reported benchmark results.The evaluation pairs complete-benchmark accuracy with multi-order probability diagnostics and paired training-recipe ablations.Source: §1 · p. 2

2 Related Work

Related work spans typed decision interfaces, audio-language models, option-order sensitivity, and consistency-based probability objectives. AudioJev adapts these strands to distributions over caller-supplied audio answers and evaluates presentation consistency separately from conventional confidence calibration.

Typed decisions and audio understanding

Option-order sensitivity

  • Prior work reports that choice ordering can change language-model decisions and probabilities, while AudioJev trains paired presentations and compares identity-aligned distributions (Pezeshkpour and Hruschka, 2024; Zheng et al., 2024).PriDe estimates option-ID priors for inference-time debiasing (Zheng et al., 2024).Source: §2 · p. 2

Consistency and calibration objectives

  • RD-SKL applies a bidirectional-KL consistency penalty to reordered options after semantic alignment; order calibration is distinct from confidence calibration against empirical correctness (Guo et al., 2017; Rahimi et al., 2020).The evaluation considers predictive accuracy and distribution sharpness alongside order calibration.Source: §2 · p. 2

Audio and conversational evaluation

3 AudioJev

AudioJev maps a waveform, question, and supplied answer list to candidate probabilities with a shared model. Its training aligns supervised views of reordered answers by meaning, while inference returns probabilities in the caller’s order.

3 AudioJev

  • A shared model maps a waveform, question, and supplied answer list to candidate probabilities, using paired training and one-forward inference.The interface supports different audio tasks through the same model.Source: §3 · p. 2, Figure 1 · p. 3

3.1 Candidate-Conditioned Decisions

  • The candidate-conditioned output assigns probabilities only to the supplied alternatives, read from next-token logits for position labels rather than generated explanations or answer sequences.The closed-set interface assumes an appropriate answer is among the supplied candidates.Source: §3.1 · pp. 2–3

3.2 Order Calibration and Semantic Alignment

  • Order calibration requires a candidate’s probability to follow its meaning under answer-list permutations, so reordered distributions must be inverse-mapped before comparison.Comparing equal label positions without that mapping would compare different answers.Source: §3.2 · p. 3

3.3 Random-Derangement SKL Training

  • Random-derangement SKL pairs views with no candidate left in its original position, supervises both distributions, and applies a symmetric-KL consistency penalty after semantic alignment.Both branches receive gradients; the paired views retain the same waveform and question.Source: §3.3 · pp. 3–4

4 Experimental Setup

The setup evaluates a shared candidate interface across semantic, acoustic, musical, and conversational decisions, using held-out data and complete MMAU and MMAR suites. Training, comparison systems, runtime, and order-calibration metrics are specified alongside predictive metrics.

4 Experimental Setup

  • The final mixture spans intent, events, scenes, speech activity, speaker transitions, sound questions, and note properties, with benchmark answers excluded from training and model selection.Evaluation includes task-specific held-out sets, 672 multilingual MINDS recordings, 3,931 short-audio MMSU questions, complete MMAU (10,000 questions), and MMAR (1,000 questions).Source: §4 · p. 4
  • AudioJev updates all active audio-classification parameters of Qwen2.5-Omni 3B with λ = 0.5 and learning rate 10−6, using three stages of 1,024, 1,024, and 2,048 updates.Each update uses four original questions and eight views; reported checkpoints use the fixed final step, and results are mean ± sample SD over three seeds.Source: §4 · p. 4
  • Comparisons include frozen direct-audio models and specialists, transcript pipelines using shared Whisper large-v3 outputs, and a single-view CE ablation matched on examples, seeds, update budgets, and final steps.The ablation removes both the second view and SKL.Source: §4 · p. 4
  • Runtime is measured for batch-one waveform-to-probabilities on a fixed five-benchmark panel, with fresh transcription included for cascades.The runtime protocol is specified in Appendix F.Source: §4 · p. 4
  • Order calibration reports SKL, total variation, and semantic-answer flips under independent random reorderings, alongside accuracy in the original supplied order.Activity uses macro F1; next-speaker prediction also uses balanced accuracy and AUROC. Position-relative questions are excluded from order-bias metrics, leaving 9,968 MMAU and 991 MMAR questions.Source: §4 · p. 4

5 Comparative Evaluation

AudioJev is evaluated across direct-audio benchmarks, shared acoustic and semantic tasks, order stability, and causal conversational decisions. It outperforms the strongest displayed transcript pipeline on complete MMAU and MMAR, while paired derangement training improves probability stability and the shared model supports diverse audio decisions.

5 Comparative Evaluation

  • The evaluation jointly examines waveform evidence, task coverage and probability stability, reporting AudioJev results as mean ± sample SD over three seeds.Fixed external references are single-run results.Source: §5 · p. 4

5.1 Direct Audio versus Transcript Pipelines

  • 68.88% on complete MMAU and 55.33% on MMAR exceed Whisper+Qwen3.5-4B by 8.14 and 3.13 percentage points, respectively.These complete-system comparisons use their respective training histories.Source: §5.1 · p. 4, Table 1 · p. 5

5.2 Shared Acoustic and Semantic Capabilities

  • Scene accuracy rises from 29.50% to 58.83% and note-property accuracy from 45.70% to 67.90% versus frozen Omni 3B; SLURP@8 gains 9.72 points.Complete MMAU accuracy is 68.88% versus 68.64% for the backbone, while Clotho-AQA remains at 90.93%; the same model also exceeds CLAP on scene accuracy and Silero on activity macro F1.Source: §5.2 · p. 5

5.3 Probability Stability under Reordering

  • 60.5% lower random-order SKL on MMAR than the single-view removal ablation accompanies answer flips falling from 18.20% to 15.44%.Across complete MMAU, MMSU and MMAR, random-order SKL is 0.0697, 0.0805 and 0.0674; Table 3 also reports semantic-flip rates of 0.70% for event/activity and 0.63% for Clotho-AQA.Source: §5.3 · p. 5, Table 3 · p. 6, Table 5 · p. 7

5.4 Conversational Decisions from Causal Audio

  • On switch tests, AudioJev’s new-session AUROC is 0.900, with a 0.096 new-session gain; evaluation spans 179 previously unused sessions, versus Smart Turn AUROCs of 0.855/0.804.AudioJev reaches 84.2%/82.3% balanced accuracy. Swapping alternatives changes semantic answers on 0.42%/0.33% of original/new-session questions, with mean TV of 0.0114/0.0103.Source: §5.4 · p. 6, Table 4 · p. 6

6 Ablations and Mechanism Analysis

The ablations test paired derangement training, SKL-weight sensitivity, and whether consistency transfers across different option orders. They distinguish probability stability from predictive correctness and hold the paired-recipe comparison conditions fixed.

6.1 Removing Paired Derangement Training

  • 43.8%, 48.9%, and 60.5% SKL reductions on MMAU, MMSU, and MMAR accompany retained accuracies of 68.88%, 61.70%, and 55.33%, respectively.The first-seed cyclic controls compare supervised reordering and CE+SKL, but are not promoted as additional final models.Source: §6.1 · p. 7

6.2 Effect and Robustness of the SKL Weight

  • Across three seeds, doubling λ from 0.5 to 1.0 changes mean accuracy by at most 0.30 points and mean flip rate by at most 0.34 points; both weights reduce SKL by 40.0–60.5%.In the single-seed sweep, λ = 0.25 gives the highest native-order accuracy, λ = 1.5 the lowest SKL, and λ = 0.75 the fewest semantic flips across the three benchmarks.Source: §6.2 · p. 7

6.3 What the Permutation Changes

  • For K ≥4, random derangements can alter adjacency beyond a single cyclic operator’s orbit; evaluation includes rotations, independent non-cyclic reorderings, and reversal.The evaluation transformations preserve waveform, question, and candidate meanings while changing the serialized answer list.Source: §6.3 · p. 7, §6.4 · p. 8
  • Entropy, maximum probability, TV, flips, and accuracy are tracked together because flatter distributions can have smaller divergence without better decisions.The analysis separates distributional stability from predictive correctness and distinguishes learned native-order behavior from an optional canonical wrapper.Source: §6.3 · p. 7

6.4 Transfer across Evaluation Orders

  • Across 12 benchmark–order conditions, RD-SKL reduces mean SKL in all; reversal reductions are 42.6% on MMAU, 47.5% on MMSU, and 58.0% on MMAR.One-step and half-cycle rotation reductions range from 38.6% to 63.2%.Source: §6.4 · p. 8, Table 7 · p. 8
  • Across all 36 seed–benchmark–order comparisons, SKL and TV decrease; across 12 benchmark–order means, absolute TV reductions range from 0.0162 to 0.0498.The two measures capture probability-ratio changes and the amount of probability mass redistributed, respectively.Source: §6.4 · p. 8
  • With examples, stage budgets, and checkpoint rules fixed, the direction holds across training draws and evaluation transformations, using caller order in one forward pass.The three seeds vary training order and fixed per-record derangements.Source: §6.4 · p. 8

7 Conclusion

AudioJev trains a shared direct-audio decision model with paired, reordered candidate views and an identity-aligned consistency objective. It reduces order divergence while retaining one-forward inference, and its weight study finds stable accuracy and flip rates when the regularization weight doubles.

Conclusion

  • AudioJev combines direct waveform classification with supervised random-derangement views and identity-aligned SKL in one shared model.The same interface covers speech, sound, music, and causal conversational decisions.Source: §7 · p. 8
  • At default λ = 0.5, order divergence decreases across audio tasks with one inference forward; doubling λ to 1.0 leaves mean accuracy and answer-flip rates stable across MMAU, MMSU, and MMAR.The conclusion describes the paired three-seed comparisons as supporting tolerance to regularization strength.Source: §7 · p. 8
  • On the fixed timing panel, one waveform question averages 62.4 ms, within a shared decision interface spanning speech, sound, music, and causal conversation.The conclusion frames the interface as carrying candidate meaning through training and inference.Source: §7 · p. 8

Limitations

The stated boundaries concern the caller-supplied closed-set interface and offline conversational evaluation. The paper distinguishes order calibration from confidence calibration and leaves deployment-specific conversational costs to the application.

Decision interface

  • AudioJev is a closed-set interface: the caller supplies admissible answers, so its stated decision setting depends on that answer list.The paper notes that identity-aligned training could extend to meaning-preserving edits such as option paraphrases.Source: Limitations · p. 9
  • Order calibration targets probability stability under option permutations, not confidence against empirical accuracy; conventional post-hoc calibrators remain applicable.The paper specifically notes that temperature scaling can be applied on top unchanged.Source: Limitations · p. 9

Conversational deployment

  • Conversational evaluation is offline at proposed pauses, independently of an agent control loop; interruption costs and waiting thresholds remain deployment-specific.The evaluated estimate concerns who speaks next from causal, past-only audio.Source: Limitations · p. 9

Ethical Considerations

The paper cautions against treating benchmark scores as validation for identity or personal-attribute decisions, and notes that dataset and model terms vary. Its task inventories and training description specify the scope and conditions of the reported evaluations.

Ethical Considerations

  • Benchmark scores do not validate personal-attribute or identity decisions; user-affecting deployment requires application-specific validation.The paper uses existing datasets and models, without recruiting participants or inferring actual mental states from expression labels.Source: Ethical Considerations · p. 9

A.1 Task Inventories and Partitions

  • The final training mixture covers seven capability families, with 8,192 questions using 7,181 waveforms and 4,167 source groups.The paper notes that dataset and model licenses vary, including research and non-commercial restrictions, and redistribution requires checking applicable terms.Source: Ethical Considerations · p. 9, §A.1 · p. 11
  • Speaker-transition tests use past-only audio at proposed pauses and measure observed next-speaker behavior, not continuous endpoint discovery.The evaluation includes original-session and new-session test windows; the latter uses previously unused sessions.Source: §A.1 · p. 11, §A.2 · p. 11

A.2 Stage Schedule

  • Across seeds, final models use the same question inventories and development partitions but vary example order and fixed per-record derangements; paired views share preprocessing.Local queries are cropped before resampling so the filter cannot use future samples.Source: §A.2 · p. 11
  • Original-example and update budgets match the removal ablation, although its single-view compute is lower; stages continue from their fixed final steps.Each original example supplies two supervised views in the final training recipe.Source: §A.2 · p. 11

B Training-Seed Replication

The reported native-order scores use the fixed final general-training step, with all three training-seed replications complete. The evaluation also reports MMAU partition accuracy and a fixed-seed weight sweep.

B Training-Seed Replication

  • All three seeds report native-order accuracy at the fixed final general-training step; Table 9 identifies the final RD-SKL models and their fixed step.The reports average each benchmark’s per-seed score, not predictions or repeated seed/question pairs; development scores did not select checkpoints.Source: B Training-Seed Replication · pp. 11–12, Table 9 · p. 12
  • Table 10 reports final-method MMAU accuracy separately for public, hidden, and full partitions, with full accuracy combining correct counts over 10,000 questions.The supplied passage identifies the scoring scope but does not provide the table’s cell values.Source: Table 10 · p. 12
  • The original-recipe removal ablation uses the same fixed general-training step as the final models.All seeds use general step 2,048 after all three stages.Source: B Training-Seed Replication · pp. 11–12
  • The broad weight sweep uses one fixed training seed, while the default λ = 0.5 and doubled weight are each evaluated with all three seeds.Replication measures variation with training order and derangements on common test partitions.Source: B Training-Seed Replication · p. 12

C Full Benchmark Scoring and External References

This block specifies the benchmark scoring scope, comparison protocols, order-calibration measurements, and implementation choices. It distinguishes native-order accuracy from wrapper diagnostics and empirical order consistency from broader guarantees.

C Full Benchmark Scoring and External References

  • Evaluation retains all 10,000 MMAU and 1,000 MMAR questions, alongside multilingual intent, MMSU, and MINDS data; benchmark answers do not enter training or checkpoint selection.MMAU combines public and hidden correct counts, while MMAR uses the publisher word-set scorer.Source: C Full Benchmark Scoring and External… · p. 12

D Complete Native and Canonical Accuracy

E Pairing-Recipe Controls

  • The fixed-seed pairing comparison tests cyclic versus random derangements and the SKL term, but only random-derangement SKL defines the final model.Small cyclic shifts preserve more relative ordering; the control does not assume they generalize better.Source: E Pairing-Recipe Controls · p. 13, Table 13 · p. 14

F End-to-End Runtime

  • The runtime panel measures fresh final-model and ASR–Laya requests across 100 questions, with transcription, reader, and end-to-end timing components.Four workers use batch size 1; assigned queries are measured three times after two warmups.Source: F End-to-End Runtime · p. 13, Table 14 · p. 14

G Order-Calibration Definitions and Implementation

  • Order calibration compares semantically aligned candidate distributions using SKL, TV, and answer flips, while duplicate descriptions are merged for semantic comparisons.SKL averages the two direct KL directions, not JS; conventional confidence-versus-correctness calibration is a separate target.Source: G Order-Calibration Definitions and… · p. 13

G.1 Training and Evaluation Permutations

  • Each record uses one fixed seed-dependent derangement for training; six shared evaluation views include random, cyclic, reverse, and canonical orders.Random-order views are not the question’s training partner, and the canonical scores are a separate wrapper diagnostic.Source: §G.1 · p. 13

G.2 Meaning-Preserving Scope

  • The bias metrics exclude questions flagged for potentially position-relative wording, but full benchmark accuracy retains them; finite-pair training does not guarantee equality for all lists or rephrasings.Ordinal alternatives and missing or paraphrased candidates are different interface interventions from the fixed-content permutations studied.Source: §G.2 · pp. 13–14

G.3 Optimization and Measurement

  • The final objective averages candidate-only CE over two views and adds SKL at λ = 0.5, updating active audio-classification parameters jointly.The inference softmax uses no target; stability metrics do not establish an application-level correctness guarantee, which remains distinct from confidence reliability.Source: §G.3 · pp. 14–15
Loading 2610.01293v1…