Source-linked AI summary
X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
Haojun Zhang, Yi Zou, Min Chen, Qize Yu, Lianrui Fan, Xini Ding, Hao Li, Shuchang Zhou, Xianming Liu, Shiyu Huang
TL;DR
Reducing audio-encoder depth can lower speech-LLM inference cost but perturbs decoder inputs and may cause deletion or premature EOS errors. X-AuT selects recoverable layer combinations with behavioral probes and restores them through progressive, cross-scale recovery; on ten benchmarks, 16 layers reduce macro error to 5.27%, while 14 layers reach 5.75% with 20.7% fewer audio-tower parameters.
Problem
Removing complete audio-encoder blocks lowers speech-LLM inference cost but perturbs decoder embeddings and can cause deletion and premature EOS errors.
Method
X-AuT uses behavioral probes for progressive layer selection, then applies representation alignment, cross-scale distillation, scheduled student-policy supervision, and LoRA recovery.
Results
Across ten benchmarks, the 16-layer model reduces macro-average error from 5.61% to 5.27%, while the 14-layer model reaches 5.75% with 20.7% fewer audio-tower parameters.
Takeaways & Limitations
The results establish 16-layer and 14-layer operating points, with accuracy varying across benchmarks and recovery depending on teacher strength, layer selection, and pruning schedule.
Takeaways & Limitations
Results are single runs from one model family, so repeated trials and broader models are needed to assess generalization.
Abstract
from arXiv · showhide
Reducing audio-encoder depth lowers the inference cost of speech large language models, but removing complete blocks perturbs the embeddings consumed by the decoder and can cause deletion and premature end-of-sequence errors. We introduce X-AuT, a progressive framework that selects layer combinations through short behavioral probes and restores the pruned model through representation alignment, cross-scale distillation, scheduled student-policy supervision, and LoRA finetuning. The language-model backbone remains frozen, while attention LoRA adapters and the tied output embedding adapt during distillation. Training uses the highest-agreement tier from a transcript-consistency pipeline, followed by source reweighting during finetuning. On ten public Chinese--English benchmarks, compressing Qwen3-ASR-0.6B from 18 to 16 audio-encoder layers reduces macro-average error from 5.61% to 5.27%. The 14-layer model reaches 5.75% with 20.7% fewer audio-tower parameters. Under the matched recipe, the 1.7B teacher yields 5.55% mean error, compared with 8.45% for self-distillation, and progressive 18$\rightarrow$14 pruning outperforms direct pruning (5.75% vs. 6.73%). These single-run results establish two practical operating points and show that the accuracy effects vary across benchmarks. Project website: https://xpeng-ai.github.io/x-aut
1 Introduction
X-AuT compresses the Qwen3-ASR audio encoder through jointly evaluated layer selection and progressive recovery, targeting lower latency while preserving decoder-compatible representations. Across ten benchmarks, 16-layer and 14-layer models provide distinct accuracy–efficiency tradeoffs.
- Motivation: Removing encoder blocks perturbs decoder conditioning embeddings and can trigger premature EOS predictions and deletion errors.The method freezes the pretrained language-model backbone while adapting decoder attention LoRA adapters and, during distillation, the tied output embedding.
- Motivation: Layer combinations must be evaluated jointly because interactions can make multi-layer recoverability differ from individual-layer importance.The paper reports that static importance scores cannot fully predict recovery under a short budget.
- Method: X-AuT selects layer combinations with short behavioral probes before applying progressive pruning and recovery under a matched training budget.The recovery combines representation alignment, cross-scale distillation, scheduled student-policy contexts, and LoRA finetuning.
- Results: The 14-layer model reaches 5.75% macro error while removing 20.7% of audio-tower parameters, establishing a higher-compression operating point.The 16-layer model reaches 5.27% macro error, while additional 14-layer losses concentrate on a few English benchmarks.
- Results: The 16-layer model remains close to or above baseline accuracy across the benchmark suite and reaches 5.27% macro error.The baseline is the unpruned Qwen3-ASR-0.6B model.
2 Related Work
Prior speech-compression methods target decoders, encoder matrices, weights, attention heads, or variable-depth training. X-AuT instead evaluates recoverability of layer combinations in a pretrained speech LLM and combines transcript filtering with mixed-context distillation.
- Existing compression: Earlier approaches include decoder reduction, low-rank encoder compression, structured sparsity, layer-drop training, and supernet-based encoder extraction.These methods target different architectural components or support variable encoder depths.
- Layer pruning: X-AuT treats recoverability as a property of a layer combination rather than an additive score assigned to individual layers.Short post-removal probes compare candidate sets under a shared recovery budget.
- Novelty: Unlike prior Whisper-layer pruning recovered with LoRA, X-AuT starts from a pretrained Qwen3-ASR audio tower and prunes it in successive hops.Its candidate probes compare layer sets under matched recovery conditions.
- Distillation: X-AuT combines teacher-forced and student-generated contexts to address the mismatch between gold-prefix supervision and inference histories.The related-work discussion positions this as a combination of two distillation regimes.
- Data selection: The training pipeline filters examples by transcript agreement and reweights sources during Stage 2 toward target-domain data.The reported experiments use the highest-agreement tier throughout recovery.
3 Method
X-AuT defines compression as selecting a recoverable ordered subset of audio-encoder layers and minimizing text error under a depth constraint. Its pipeline filters supervision, probes candidates, and recovers each student through three stages combining alignment, distillation, and finetuning.
- Problem formulation: The method seeks an ordered layer subset and adapted parameters that minimize aggregate text error under a target depth smaller than the original encoder.The decoder backbone remains frozen while selected decoder-side parameters can adapt.
- Data preparation: Transcript-consistency filtering compares a source transcript with two ASR hypotheses and assigns one of nine agreement tiers using pairwise edit rates.The reported configuration uses class 1, the highest-agreement tier, in both distillation stages.
- Candidate selection: The encoder is pruned progressively from 18 to 16 to 14 layers, with matched single-layer and pair probes selecting candidates by development-suite TER.The 16-to-14 hop selects layers {5, 6}; pair probes are used because multi-layer recovery is not reliably predicted by single-layer scores.
- Stage 0: Representation Alignment: Stage 0 aligns intermediate, bridge, and logit representations with teacher-forced transcript supervision before later distillation and finetuning.Teacher and student widths are reconciled with grouped layer matching and a learned bottleneck projection.
- Stage 1: Distillation: Stage 1 combines bridge alignment, teacher-forced logit distillation, gold-transcript CE, and scheduled student-policy supervision.Every fifth optimizer step uses student-generated prefixes after 20% of Stage 1, while rollout safeguards can fall back to teacher-forced objectives.
- Stage 2: LoRA Finetuning: Stage 2 finetunes the audio encoder, bridge, and decoder LoRA adapters with gold-transcript CE while reweighting class-1 sources toward target domains.This stage contains no teacher loss and freezes the tied output embedding.
4 Experimental Setup
The experiments use standardized multilingual ASR data, transcript-consistency filtering, fixed development suites, and full public evaluation, with single-run checkpoint reporting and separate efficiency measurements.
- The source pool exceeds 280k hours across public and proprietary multilingual ASR corpora, with audio capped at 40 seconds.
- Transcript records are normalized into unified manifests, filtered using dual-system transcription and consistency labels, and the reported runs use the highest-agreement class.
- Checkpoint selection uses five fixed development subsets and their unweighted macro average, while final results use ten public benchmarks with equal benchmark weighting.
- The 0.6B runs use 32 accelerators, global batch size 512, bf16 training, and a three-stage schedule totaling 0.05, 0.95, and one epoch.
- Results are single-run observations without repeated seeds or bootstrap confidence intervals, and efficiency reports are descriptive averages without retained run-to-run variability.
5 Results
Across the evaluated settings, progressive recovery produces two accuracy–efficiency operating points, while teacher strength, joint layer selection, and pruning schedule materially affect outcomes.
- 5.1 Main Results: 5.27% macro error is achieved by the 16-layer model versus 5.61% for the baseline, while the 14-layer model reaches 5.75% with 20.7% fewer audio-tower parameters.
- 5.2 Training Trajectories: 5.36% selection-suite TER follows Stage 2 finetuning from the 5.76% Stage 1 minimum, after rapid early recovery and slower subsequent optimization.
- 5.3 Teacher-Scale Control: 5.55% mean error with the 1.7B teacher beats 8.45% with self-distillation across all ten benchmarks under the matched recipe.
- 5.4 Behavior-Driven Layer Selection: The {5, 6} pair reaches 6.93% TER versus 7.78% for {6, 8}, showing that pair interactions are not predicted by individual-layer scores alone.
6 Discussion
X-AuT’s results indicate that recoverability depends on teacher strength, joint layer selection, and progressive pruning, while alignment cannot fully replace capacity lost at deeper compression. Interpretation is limited by single-run evaluation, restricted model coverage, and incomplete end-to-end efficiency measurement.
- Interpretation: Representation alignment and retained capacity jointly shape recovery: Stage 0 addresses intermediate and bridge mismatch, while Stage 1 supervises token distributions under gold and student-generated prefixes.Alignment lets the 16-layer model surpass baseline macro error, but the 14-layer gap indicates it cannot fully replace lost capacity.
- Teacher scale: 5.55% mean error with the 1.7B teacher versus 8.45% with self-distillation shows that the recovery signal materially affects the tested outcome.The improvement spans all ten benchmarks and is particularly pronounced on CommonVoice zh/en.
- Pruning schedule: 5.75% mean error with progressive 18→16→14 pruning versus 6.73% with direct 18→14 pruning favors staged compression under the matched nominal budget.The 16-layer model reaches 5.27% macro error, while the 14-layer model reaches 5.75%.
- Layer selection: Joint pair probes improve selection beyond individual-layer rankings: the selected {5, 6} pair outperforms jointly pruning the two strongest single removals, L8 and L6.This supports evaluating layer selection and recovery together rather than relying on static importance scores.
- Limitations: Single-run seed-42 results lack repeated-seed variation, confidence intervals, and significance tests, making small differences descriptive observations.Fixed development subsets capped at 25 utterances per benchmark introduce selection noise, including around the 0.14-pp 14-layer versus baseline gap.
- Scope: The study covers one model family and limited pruning candidates, while efficiency averages omit run-to-run uncertainty and only the audio tower is compressed.Broader model-family, candidate, data-mixture, and repeated hardware evaluations are identified as needed to assess generalization and deployment impact.
7 Conclusion
X-AuT compresses the Qwen3-ASR-0.6B audio tower through progressive pruning and cross-scale recovery, establishing 16- and 14-layer operating points. The results also associate recovery quality with teacher strength, joint layer selection, and pruning schedule, but broader models and repeated trials remain necessary.
- Conclusion: 5.27% macro-average error follows compression from 18 to 16 layers, while 14 layers reaches 5.75% with 20.7% fewer audio-tower parameters.The controls associate recovery with teacher strength, joint layer selection, and pruning schedule.
A.1 Full Hyperparameter Configuration
The appendix identifies Table 4 as the main 16-layer hyperparameter reference and states that the 14-layer run reuses the same recipe from the recovered 16-layer checkpoint.
- A.1 Full Hyperparameter Configuration: Table 4 lists the main 16-layer hyperparameters, while the 14-layer run uses the same recipe and recovered 16-layer initialization.
A.2 Premature-EOS Safeguards
X-AuT addresses premature EOS after pruning with tied-head adaptation, minimum-token gating, and rollout filtering. The combined safeguards achieve the best observed TER, while gating removes empty events at the cost of more rejected rollouts.
- A.2 Premature-EOS Safeguards: The safeguards target shifted bridge conditioning embeddings that can weaken acoustic evidence and trigger EOS after only a few tokens.The paper presents this as a hypothesis rather than a mechanistic proof.
- A.2 Premature-EOS Safeguards: Minimum-token gating removes empty events, but increases filter rejections to 2113–2162 compared with 11–20 without gating.The no-mitigation condition has five windows with nonzero empty ratio, with a maximum of 0.3% and mean of 0.018%.
- A.2 Premature-EOS Safeguards: 6.75% best observed TER occurs with combined safeguards, versus 6.83% for tied-head-only and 7.58% after drift without mitigations.The combined configuration also has mean rollout length 15.06 tokens versus 11.05 for the no-mitigation condition.
A.3 Teacher-Scale Comparison
Cross-scale supervision substantially improves the 16-layer student over self-distillation under matched settings, while progressive pruning and the retained hybrid recipe show distinct trade-offs in recovery and efficiency.
- Teacher-scale comparison: 5.55% mean error with the 1.7B teacher versus 8.45% with self-distillation, with the cross-scale checkpoint better on every public benchmark.The comparison uses matched Stage 0/1 settings and single-run best development checkpoints.
- Teacher-scale comparison: The teacher comparison remains a single-run result with projection modules used only when teacher and student dimensions differ.The authors therefore do not treat it as proof that the teacher creates capabilities absent from every unpruned student.
- Pruning schedule comparison: 5.75% mean error for progressive pruning versus 6.73% for direct 18→14 pruning, with progressive pruning better on all ten benchmarks.Both approaches remove the same original layers and use the same nominal recovery and data budget.
- On-policy strategy comparison: 6.41% best TER for hybrid on-policy training versus 6.98% for plain off-policy, but matched off-policy performs best at 6.23%.Hybrid training is retained because it directly supervises student-generated contexts and performs well in the complete pipeline.
- Inference efficiency: 21.4% lower encoder time on the in-vehicle PPU and 11.4% lower time on H800 do not translate into similarly large end-to-end gains.End-to-end reductions are 4.7% and 2.6%, respectively, because autoregressive decoding dominates total time.