Source-linked AI summary
Mizar: A 159M-Parameter Audio-Language Model for Audio Understanding
Kaiyang Li, Shaobo Han, Yue Tian, Shihao Ji
TL;DR
Practical local audio understanding motivates compact ALMs that fit limited memory and computation budgets. Mizar combines a compact audio-language architecture with diverse supervision and three-stage training; it surpasses the previous best-performing ALM below 200M parameters across three benchmarks and runs on a single CPU.
Problem
Local audio understanding motivates ALMs below 200M parameters that fit the memory, compute, and energy budgets of affordable hardware.
Source: §1 · p. 1Method
Mizar is a 159.3M-parameter ALM combining a CED-Small encoder, frequency-merging mapper, SmolLM2-135M decoder, diverse audio supervision, and three-stage training.
Source: §1 · p. 1Results
Takeaways & Limitations
Mizar demonstrates strong audio understanding under a compact parameter budget and supports local inference on a single CPU.
Source: §4 · p. 4Mellow’s published 52.11% MMAU Test result is from an earlier test and is not directly comparable across benchmark revisions and scoring protocols.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Audio-language models (ALMs) integrate acoustic perception with the knowledge encoded in language models, enabling contextual understanding of auditory events. Making these capabilities practical on devices with limited memory and computation motivates our focus on small ALMs with fewer than 200M parameters. We introduce a recipe that brings together architecture, data, and three-stage training to build Mizar, a 159.3M-parameter ALM. Its architecture connects a compact CED-Small audio encoder to SmolLM2-135M through a frequency-merging mapper. With supervision drawn from ReasonAQA, AudioMCQ, and AVQA, the model undergoes three training stages: audio-language alignment (Stage 1), audio-dependent fine-tuning (Stage 2), and post-training (Stage 3) aimed at strengthening weak skills while retaining learned capabilities. Across five random seeds, Mizar achieves mean accuracies of 52.92% on MMAU, 42.42% on MMAR, and 36.02% on ADQA-clean, surpassing the previous best-performing ALM below 200M parameters on all three benchmarks. It also supports local inference on a single CPU: on questions from the MMAU benchmark, the mean latency from opening the audio file to generating a complete answer is 1.09 seconds. Code and checkpoints are available at https://github.com/KaiyangLi1992/Mizar_159M.
1. INTRODUCTION
Practical local audio understanding calls for ALMs below 200M parameters, while Mellow provides a strong starting point for further optimization. Mizar combines a compact architecture, expanded supervision, and staged training, with reported gains across benchmarks and CPU inference.
1. INTRODUCTION
- Compact ALMs are motivated by local audio understanding on commodity CPUs and modest edge processors, where memory, compute, and energy budgets are constrained.Local inference can support continuous operation without reliable connectivity while keeping sensitive recordings on-device.Source: §1 · p. 1
- Mellow’s published 52.11% is an earlier MMAU Test result and is not directly comparable across benchmark revisions and scoring protocols.Mellow’s 167M-parameter result motivates further engineering optimization across data, training, and architecture.Source: §1 · p. 1
- Mizar is a 159.3M-parameter ALM connecting a frozen CED-Small encoder to SmolLM2-135M through a frequency-merging mapper, trained with ReasonAQA, AudioMCQ, and AVQA supervision.Its three stages align audio and language, strengthen reliance on acoustic evidence, then refine skills with rehearsal and answer-position balancing.Source: §1 · p. 1, Figure 1 · p. 2
- Mizar reports relative accuracy gains over Mellow of 27.61% on MMAU, 23.31% on MMAR, and 19.07% on ADQA-clean, and answers MMAU questions on one CPU in 1.09 seconds on average.Its 52.92% mean accuracy on MMAU across five random seeds also exceeds several 7B–13B references.Source: §1 · p. 1
2. ARCHITECTURE, DATA, AND TRAINING
Mizar maps frozen audio-encoder features into the language decoder’s embedding space, then conditions answer generation on audio and question text. Its training progresses from audio-language alignment through audio-dependent fine-tuning to a refinement stage that balances answer positions and rehearses learned skills.
2.1. Model architecture
- The mapper concatenates four frequency features at each of 126 time positions, projects them to 576 dimensions, adds global and positional context, and refines them into audio tokens for the decoder.The encoder feature grid is 126 × 4 × 384; the mapper produces a 126 × 576 audio-token sequence.Source: §2.1 · p. 2
- SmolLM2-135M receives the audio tokens, a separator, and tokenized question and option text, then generates the answer autoregressively.The CED-Small encoder remains frozen while the mapper is trained and the language decoder is fine-tuned.Source: §2.1 · p. 2, §2.2 · p. 2
2.2. Data and three-stage training
- Stage 1 combines ReasonAQA and AudioMCQ examples, including sampled chain-of-thought data, to align audio and language and adapt the decoder to audio tasks.Stage 2 fine-tunes on strongAC examples selected for low reference-model accuracy when audio is replaced by silence, plus AVQA sound and audio–visual subsets.Source: §2.2 · p. 2
- Stage 3 cyclically reorders options to balance correct-answer positions and uses teacher–student four-quadrant sampling to target student weaknesses while retaining previously learned skills.The four groups are based on whether the teacher and Stage 2 student answer correctly under all four cyclic option orderings; 12,800 examples are used.Source: §2.2 · p. 3, Table 2 · p. 3
3. EXPERIMENTS AND ANALYSIS
Mizar improves on Mellow across the three benchmarks and in measured compute, while audio-input diagnostics show a larger accuracy drop for Mizar when original audio is removed. The experiments also examine training, mapper, and encoder choices, then measure CPU inference.
3.1. Experimental setup
- The evaluation uses MMAU, MMAR, and ADQA-clean, greedy decoding, and an unweighted three-benchmark mean, with five-seed mean ± sample SD for post-training results.MMAU test-mini is the validation set for selecting hyperparameters and checkpoints; an audio-level audit found no overlap among training, validation, and test sets.Source: §3.1 · p. 3
3.2. Benchmark results
- Mizar gains 11.45% on MMAU Test, 8.02% on MMAR, and 5.77% on ADQA-clean over locally reevaluated Mellow; Stage 3 reaches a 43.78% Test-3 mean.The benchmark comparison reports improvements across all three benchmarks; the Stage 3 result is a mean across five seeds.Source: §3.2 · p. 3, Table 3 · p. 3
3.3. Ablation on Stage 1 and 2 Components
- Stage 2 fine-tuning on strongAC and AVQA raises Test-3 from 40.10% to 42.38%; CED-Small achieves a higher mean than HTS-AT and BEATs.Removing AudioMCQ CoT examples lowers Test-3 by 2.19 points, and using ReasonAQA alone lowers it by 6.79 points; B1/B2 encoder comparisons also change the mapper.Source: §3.3 · p. 3, Table 4 · p. 3
- The 3.37M-parameter mapper reaches a 40.01% Test-3 mean, versus 39.27% for Concat and 39.10% for Mean.On MMAU, the mapper reaches 49.03%, compared with 46.93% for Concat and 46.94% for Mean; these are single-seed runs.Source: §3.3 · p. 3, Table 4 · p. 3
- Compared with the released Mellow checkpoint (35.37%), MMAU and MMAR accuracies increase from 41.47% to 48.21% and 34.40% to 38.60%, respectively, while ADQA-clean remains nearly unchanged (30.25% versus 30.31%).Source: §3.3 · p. 4
3.4. Post-training analysis
- Stage 3 reaches a 43.78% Test-3 mean; Q2 outperforms R2 by 0.51, 0.86, and 0.88 points on MMAU, MMAR, and ADQA-clean, respectively.Q2 combines quadrant selection with answer-position balancing; R2 balances answer positions on the same questions.Source: §3.4 · p. 4, Table 5 · p. 4
3.5. Dependence on acoustic input
- Silence and replacement audio reduce accuracy on every benchmark, with larger drops for Mizar than Mellow; Mizar remains more accurate under both interventions.On MMAU, the drops are 8.23 and 9.24 points for Mizar, versus 5.80 and 6.51 points for Mellow.Source: §3.5 · p. 4, Table 6 · p. 4
3.6. Computational cost and CPU inference
- Mizar uses 113.22 GFLOPs per input versus Mellow's 118.95, a 4.82% decrease; its complete-answer CPU latency averages 1.094 seconds.The audio–language interface accounts for 73.06% of the compute saving; CPU measurements use four threads, FP32, and no quantization.Source: §3.6 · p. 4, Table 7 · p. 4
4. CONCLUSION
Mizar combines compact audio architecture, diverse supervision, and staged adaptation to outperform the previous best-performing ALM below 200M parameters across three benchmarks.
4. CONCLUSION
- Mizar surpasses the previous best-performing ALM below 200M parameters across MMAU, MMAR, and ADQA-clean.The conclusion describes strong audio understanding under a compact parameter budget.Source: §4 · p. 4
- Audio instruction learning and audio-dependent fine-tuning establish the foundation, which answer-position balancing and teacher-assisted selection further strengthen.Source: §4 · p. 4
- Mizar’s recipe brings together a compact audio architecture, diverse supervision, and staged adaptation.Source: §4 · p. 4
- Mizar runs on a single CPU, making local inference feasible with limited computing resources.Source: §4 · p. 4