Source-linked AI summary
Correlation-Based Distillation Yields More Mergeable Speech-Music Encoders
Fabian Ritter-Gutierrez
TL;DR
Prior work fixed the distillation loss when merging speech and music students, leaving open whether the loss affects mergeability. This paper compares LKD with correlation-based LCL under two post-hoc merging algorithms and a controlled evaluation. Across both algorithms, LCL improves speech-side mergeability, suggesting the effect is associated with the representation it produces.
Problem
Prior studies fixed the distillation loss in post-hoc merging, leaving open whether loss choice changes how well distilled speech and music students merge.
Source: §I · p. 1Method
The paper substitutes correlation-based LCL for LKD while holding the student pool and nine-task evaluation fixed across weight-interpolation and activation-permutation merging.
Source: §III · p. 2Results
Across both post-hoc algorithms, LCL improves mergeability; under interpolation, its students stay closer to their better endpoint in 17 of 18 comparisons and lead on speech.
Source: §VIII · p. 5Takeaways & Limitations
The shared loss ordering across two algorithms with no common mechanism points to the student representation as a contributor, while LKD retains the higher standard-width Music Score.
Source: §VI · p. 4, §VII · p. 5Whether the loss effect transfers to TIES-Merging, DARE, or speech-specific merging remains open.
Source: §VII · p. 5
Abstract
from arXiv · showhide
A single compact encoder for both speech and music removes the need to maintain a separate model per domain. A practical recipe distils a speech teacher and a music teacher into two small students, then merges them into one model. Two merging algorithms exist: interpolating the student weights from a shared initialisation, or permuting their channels into alignment before averaging. Prior work fixed the distillation loss in both, leaving open whether the loss itself affects how well the students merge. We show that it does. Holding architecture and evaluation fixed, we replace the DistilHuBERT loss \Lkd{} with a correlation-based loss \Lcl{}. Under weight interpolation, \Lcl{} students stay closer to their own better endpoint in $17$ of $18$ comparisons and lead on the speech tasks at every interpolated weight. Under activation permutation, they need fewer channels rearranged at three layers, their matched channels correlate more strongly, and they again lead on speech. The two merging algorithms share no mechanism, yet both improve under the same loss substitution. This suggests that the gain lies in the representation \Lcl{} produces rather than in either algorithm.
I. INTRODUCTION
The paper asks whether distillation loss affects how well speech and music students merge, a question left open by prior work that fixed the loss. It substitutes correlation-based LCL for LKD and evaluates both post-hoc merging algorithms under a fixed setup.
Research gap
- Prior task-arithmetic work fixed the distillation loss, while correlation-permutation had only been applied to large pre-trained encoders, leaving its use for distilled students untested.The paper therefore asks whether merge-time behavior depends partly on the student produced by the loss, rather than solely on the merging algorithm.Source: §I · p. 1
Approach
- The study holds the students and nine-task evaluation constant while replacing standard LKD with correlation-based LCL across two merging algorithms.The algorithms operate respectively in weight space from a shared initialisation and by aligning representation activations.Source: §I · p. 1
Contributions
- LCL students stay closer to their better endpoint in 17 of 18 task-arithmetic comparisons and lead on Speech Score at every interpolated weight.The paper also reports that LCL students need less channel rearrangement, have stronger channel matches, and lead on speech under correlation-permutation.Source: §I · p. 1
- The new work evaluates LCL merges under both algorithms, permutation alignment statistics on distilled students, a wide configuration, and nine-task rescoring.The task-arithmetic study used LKD only; the paper identifies these evaluations as its additions.Source: §I · p. 1
II. BACKGROUND
The background reviews two post-hoc merging methods and the distillation objectives compared in the paper. Task arithmetic interpolates student weights, while correlation-permutation aligns activations before averaging.
Task arithmetic
- Task arithmetic forms speech and music task vectors from students trained from a shared initialisation, then interpolates those vectors with λ1 trading speech against music.The task vectors are each student’s weights minus the shared initialisation.Source: §II · p. 1
Correlation-permutation
- Correlation-permutation uses per-layer linear assignment to maximize activation cross-correlation before averaging, without requiring shared initialisation.It permutes selected components, including attention heads, within-head channels, the second FFN projection, and the CNN front end; query, key, and value projections remain untouched.Source: §II · pp. 1–2
Distillation loss
- LCL replaces DistilHuBERT’s layer-prediction loss LKD with a correlation-based objective, while retaining a scaled cosine term.Its cross-correlation term drives student–teacher feature correlations toward identity, and its self-correlation term decorrelates student features.Source: §II · p. 2
Ensemble distillation
- Ensemble distillation trains one student simultaneously from both teachers, avoiding post-hoc interpolation and serving as a λ-invariant baseline.Both teachers provide gradients during training.Source: §II · p. 2
III. MERGEABILITY PROTOCOL
The protocol isolates the effect of swapping LKD for LCL by keeping the student pool and nine-task evaluation fixed. It assesses mergeability through interpolation robustness, structural alignment, and merged-model quality.
Controlled comparison
- The protocol applies the same shared-initialisation students to both merging algorithms and scores them under one nine-task evaluation, making the loss swap the sole difference.The task suite covers four SUPERB speech tasks, four MARBLE music tasks, and ESC-50 environmental sound classification; three student configurations are tested.Source: §III · p. 2
Scoring and configurations
- Every student shares the listed configuration, and each task score is normalized between an FBANK floor and the better teacher.Speech Score averages four speech tasks; Music Score averages five music and audio tasks; Average Score averages those domain scores.Source: Table I · p. 2, §III · p. 2
Measures
- Mergeability has three distinct measures: endpoint-relative mean deviation for interpolation robustness, moved-channel counts and matched correlations for structural alignment, and Average Score for merge quality.The three measures need not move together.Source: §III · p. 2
IV. ANALYSIS OF LCL ON TASK-ARITHMETIC INTERPOLATION
Under task arithmetic, LCL is more robust than LKD in nearly all reported comparisons, and its speech scores are consistently higher across interpolated settings. The wide configuration also shows that the speech-side robustness margin can be small when initialization is controlled.
Wide configuration
- At λ1=0.8, the wide pair’s music deviations are −1.32 pp for LCL and −2.10 pp for LKD, while speech deviations are near-tied at −1.38 pp and −1.56 pp.Both wide students start from a shared random checkpoint, removing the HuBERT-initialisation confound; the authors caution that the speech margin is too small to attribute the robustness gain to loss alone.Source: §IV · pp. 2–3
Task-arithmetic robustness
- 17 of 18 task-arithmetic comparisons favor LCL for endpoint-relative robustness; the exception is two-layer speech at λ1=0.9, where LCL is −0.30 pp against LKD’s −0.63 pp.At three layers and λ1=0.8, LCL’s mean deviation is −0.59 pp against −4.51 pp on music and −1.01 pp against −3.01 pp on speech.Source: §IV · p. 2, Table II · p. 3
Task scores
- LCL has the higher Speech Score in all twelve interpolated cells, reversing only at the wide pair’s music-only endpoint by 0.9 points.At three layers and λ1=0.8, LCL reaches 959.4 Speech Score against 924.1, with IC 93.57% against 89.24% and ASR 13.62% against 17.08% WER.Source: §IV · p. 3
Per-task analysis
- At λ1=0.8, the two-layer speech mean deviation is −2.33 pp for LKD and −1.80 pp for LCL, with degradation spread across tasks and largest on ASR.Per-task advantages are significant on ASR at λ1=0.8 and 0.5 for the two-layer pair and on IC at those weights for the three-layer pair; composite scores and mean deviations were not tested.Source: §IV · p. 3
V. ANALYSIS OF LCL ON PERMUTATION MERGING
Under correlation-permutation, LCL students need less channel rearrangement and have more strongly matched channels than LKD students, while leading on Speech Score. The structural advantage is strongest in early FFN and convolutional layers, with some music-task trade-offs.
V. ANALYSIS OF LCL ON PERMUTATION MERGING
- At three layers, first- and second-block FFN permutation rates fall from 98.2% to 78.3% and from 99.3% to 89.8% under LCL; by block three, both require near-complete reordering (99.5% versus 98.5%).The reduction is concentrated where the students’ FFNs first diverge; attention rates drop less.Source: §V · p. 3
- For three-layer students, LCL requires 21.3% of first-convolutional-layer channels reordered versus 42.0% under LKD, and none in layers four and five versus up to 33.6%.The authors suggest that retaining more of the shared front end may explain slightly lower music-task scores, because it remains closer to the speech-initialised starting point.Source: §V · p. 3
- Mean matched absolute channel correlation is 68.6% versus 60.3% for two-layer LCL and LKD students, and 69.3% versus 59.9% at three layers.These alignment statistics do not involve a downstream probe; independent initialization lowers the two-layer LKD mean from 60.3% to 47.0%.Source: §V · p. 3
- LCL has the higher Speech Score at both CP weights and depths: 957.1/952.5 and 933.1/923.0 at two layers, and 964.6/958.5 and 953.9/927.8 at three.Average Score follows in three of four comparisons; the exception is three layers at λ1=0.9 (939.2/941.4).Source: §V · p. 3
VI. CONSISTENCY ACROSS MERGING ALGORITHMS
Across task arithmetic and correlation-permutation, LCL leads on Speech Score in all thirteen comparisons, though Average Score and individual-task results are less uniform. The authors associate the cross-algorithm agreement with the student representation, while noting that the proposed self-correlation mechanism remains only partly tested.
VI. CONSISTENCY ACROSS MERGING ALGORITHMS
- Of 74 testable per-task comparisons, 12 reach significance, all with |h| < 0.13 on classification tasks; 16 of 90 comparisons lack prediction files.Four of five configurations show no significant differences at λ1=0.9.Source: §VI · p. 4
- Because TA and CP share no merging machinery, their agreement points to the representation shaped by the loss, although the shared probe suite could contribute to that agreement.Permutation rates and matched correlations are probe-free structural evidence; the authors hypothesize that self-correlation decorrelates feature dimensions, but only the feature-space reorganization is established.Source: §VI · p. 4
- LCL has the higher Speech Score in all thirteen comparisons across task arithmetic and correlation-permutation, and the higher Average Score in ten.The exceptions for Average Score are two-layer task-arithmetic merges at λ1=0.9 and 0.8, and the three-layer CP merge at 0.9.Source: §VI · p. 3, Figure 1 · p. 4
- Whether self-correlation-driven feature reorganization improves mergeability remains untested because the companion ablation performs no merging.The authors leave a λsc sweep followed by re-merging for future work.Source: §VI · p. 4
VII. DISCUSSION
The discussion weighs the practical trade-offs among merging methods and shows that performance depends on configuration and domain priority. Task arithmetic and correlation-permutation diverge most in the wide configuration, while transfer of the reported loss effect to other merging methods remains open.
VII. DISCUSSION
- At λ1=0.8, task arithmetic exceeds correlation-permutation by 14.0 points for the wide pair and 8.8 points for the two-layer LKD pair.At λ1=0.9, the two post-hoc methods track each other closely.Source: Figure 2 · p. 5
- At standard width, LKD retains the higher Music Score at every weight, including 935.1 against 911.0 at the music-only endpoint.LCL improves both post-hoc methods on the speech side, so the preferred loss depends on domain priority and practical constraints.Source: §VII · p. 5
- At λ1=0.9 all ten post-hoc merges beat their architecture’s ensemble baseline, but at 0.8 four fall below it.Both standard-width LCL pairs remain above the baseline at 0.8.Source: §VII · p. 5
- For the wide D=1536 students, correlation-permutation still requires 99.9–100% transformer permutation, while its six 256-channel heads make alignment coarser than the 12 64-channel heads in the standard student.The discussion presents this as a plausible factor favoring task arithmetic, not as an established explanation for transformers.Source: §VII · p. 5
- Whether the loss effect transfers to TIES-Merging, DARE, or speech-specific merging remains open.Correlation-permutation removes the shared-initialisation requirement, but the two independently initialised students score 815.5 at λ1=0.8, with IC falling from 93.99% to 65.49%.Source: §VII · p. 5
VIII. CONCLUSION
The distillation loss is not merge-neutral: replacing LKD with LCL improves speech-side merging under both tested algorithms, while LKD retains the standard-width music advantage. The agreement across algorithms points to the representation produced by LCL, rather than either algorithm, as the more plausible source of the gain.
VIII. CONCLUSION
- Under weight interpolation, LCL stays closer to its own better endpoint in 17 of 18 comparisons and leads on Speech Score at every interpolated weight; under permutation, it needs fewer rearrangements and leads on speech.LKD retains the higher Music Score at standard width; agreement across algorithms with no shared mechanism points more plausibly to the representation LCL produces.Source: §VIII · p. 5