PaperlayerView exact revision on arXiv

Source-linked AI summary

ConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence

Ming-Hao Hsu, Shujie Liu, Yuxuan Hu, Yuxiang Wang, Yingda Shen, Jinyu Li, Zhizheng Wu

arXiv:2610.23539v1cs.CLcs.SDeess.AS

TL;DR

Speech and audio models must connect diverse acoustic information to text-pretrained decoders, and their interfaces determine how acoustic representations relate to text embeddings. ConvexLLM uses continuous convex combinations of frozen text embeddings and leads the compared models on VoiceBench, with task-dependent tradeoffs in audio understanding and transcription.

  • Problem

    Speech and audio models must carry diverse acoustic information into pretrained language decoders, including spoken content and cues absent from transcripts.

    Source: §1 · p. 1

  • Method

    ConvexLLM uses C-GATE to represent audio as convex combinations of frozen text embeddings in a general-purpose speech and audio model.

    Source: §1 · pp. 1–2

  • Results

    ConvexLLM leads the compared models on the eleven-task VoiceBench aggregate, with its largest margins on spoken knowledge and reasoning, while baselines lead on MMAU and transcription.

    Source: §7 · pp. 7–8

  • Takeaways & Limitations

    The results support convex combinations of text embeddings as a viable interface for general-purpose speech and audio language models.

    Source: §7 · p. 8

    The main comparison spans systems differing in scale, training data, and adaptation recipe, so it compares complete systems rather than interfaces in isolation.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Speech and audio language models connect continuous acoustic representations to decoders pretrained primarily on text data. Most such models use MLP or Q-Former interfaces to map acoustic features into the decoder's input space, requiring the decoder to learn how to interpret the resulting representations. We investigate whether grounding these representations in the decoder's existing text embeddings can help it use acoustic information while better preserving its pretrained text capabilities. We present ConvexLLM, a general-purpose speech and audio language model whose C-Gate interface represents audio as continuous convex combinations of frozen text embeddings. ConvexLLM achieves an eleven-task VoiceBench aggregate of 75.36, the highest among the open-weight models we compare, and an MMAU Test-mini accuracy of 62.50%. Its largest margins over the baselines are on spoken knowledge and reasoning tasks, while the strongest baselines are more accurate on audio understanding and transcription. In a comparison with a 2B-parameter decoder, C-Gate achieves higher aggregate spoken-language performance than MLP and Q-Former interfaces, with text-task performance close to MLP and task-dependent tradeoffs in transcription and audio understanding. These results support convex combinations of text embeddings as a viable interface for general-purpose speech and audio language models.

1 Introduction

Speech and audio models must connect diverse acoustic information to text-pretrained decoders, while their input interfaces shape how that information is represented. The introduction presents CONVEXLLM’s convex text-embedding interface and motivates evaluating it across speech and audio tasks.

1 Introduction

  • Audio models must convey spoken content as well as paralinguistic, environmental, or musical cues, including information absent from transcripts.Spoken requests also draw on the language model’s knowledge and instruction-following capabilities.Source: §1 · p. 1
  • C-GATE represents audio at each position as a convex combination of selected frozen text embeddings, keeping outputs inside their convex hull.Non-negative weights that sum to one define the convex combination; Figure 1 contrasts this with discrete and unconstrained interfaces.Source: §1 · p. 1, Figure 1 · p. 2
  • The evaluation tests C-GATE in a complete model across broader speech and audio demands than prior controlled recognition and emotion-recognition settings.The comparison with MLP and Q-Former uses shared Stage 3 data, objectives, and effective-batch optimization settings.Source: §1 · pp. 1–2
  • C-GATE has its strongest spoken-language performance, while transcription and non-speech audio tasks favor different interfaces.The introduction also notes that text-task performance is close to MLP.Source: §1 · p. 2
  • CONVEXLLM achieves state-of-the-art VoiceBench performance among comparable-scale open-weight audio and omni models, driven by spoken knowledge and reasoning.Its eleven-task aggregate is higher than those of MiniCPM-o 4.5, Kimi-Audio, and MiMo-Audio.Source: §1 · p. 2

2 Related Work

Prior work connects audio to language decoders through learned interfaces and uses training strategies to support language capability retention. This paper extends vocabulary-based interfaces to broad speech and audio tasks and examines text performance after adaptation.

Audio Language Models

  • Open-weight audio models combine audio encoders or tokenizers with pretrained language decoders, but unconstrained interfaces can produce representations far from text embeddings.Adapting language models to speech can also degrade existing text capabilities.Source: §2 · p. 2

Language Capability Transfer and Retention

  • Prior retention strategies use text data, distillation, simulated recognition posteriors, or text-referenced rewards; this work instead constrains speech representations to frozen text embeddings’ convex hull.The paper measures post-adaptation text performance and the gap between spoken and written versions of the same tasks.Source: §2 · pp. 2–3

Vocabulary-Based Interfaces

  • Earlier vocabulary-based studies had not demonstrated a general-purpose model spanning speech recognition, spoken reasoning, instruction following, and non-speech audio understanding.CONVEXLLM builds on C-GATE and evaluates text performance after audio adaptation.Source: §2 · p. 3

3 CONVEXLLM

CONVEXLLM connects an acoustic encoder to a pretrained language decoder through C-GATE, which represents audio using convex mixtures of frozen text embeddings. Its training objective combines response generation with text-teacher distillation for audio examples that have written counterparts.

3 CONVEXLLM

  • CONVEXLLM combines an acoustic encoder, C-GATE, and a pretrained language decoder, then adapts them through staged multitask training.Audio inputs and task prompts condition the decoder to generate textual responses.Source: §3 · p. 3, §3.1 · p. 3

3.1 Problem Formulation

  • The pretrained text embedding table stays frozen, providing a fixed reference while the interface learns to combine its rows for pooled audio states.The encoder’s temporal mean-pooled states are mapped to decoder inputs.Source: §3.1 · p. 3

3.2 Convex Routing

  • C-GATE scores vocabulary keys for each audio-position query, selects the top-K embeddings, and mixes them with non-negative weights that sum to one.The resulting interface output lies within the convex hull of the frozen embeddings.Source: §3.2 · p. 3
  • C-GATE selection is not differentiated; gradients reach the router through the scores of selected embeddings.The learned routing temperature adjusts mixture sharpness, while the mixture temperature rescales it by a fixed factor.Source: §3.2 · p. 3

3.3 Decoder Conditioning and Training Objective

  • For audio with a written counterpart, the model matches a frozen text teacher’s next-token distributions; other audio and text-only examples use cross-entropy.The teacher reads the spoken content in written form and is used only during training; inference uses the audio-conditioned model alone.Source: §3.3 · p. 4

4 Experimental Setup

The experiments train and evaluate CONVEXLLM on speech, spoken-instruction, and non-speech audio tasks, alongside text replay examples. They use staged training, compare complete open-weight systems, and report results on three benchmarks.

4.3.1 Training Data

  • Training combines public speech and audio datasets with synthetic spoken instruction and reasoning data, providing multitask supervision across speech and non-speech tasks.A small proportion of text-only examples, called text replay, is included to help preserve the decoder’s text capabilities.Source: §4 · p. 4, §4.3.1 · p. 5

4.1 Models and Implementation Details

  • The model uses Whisper-large-v3 as its acoustic encoder and Qwen3.5-9B as its main decoder; the 2B interface comparisons use Qwen3.5-2B.C-GATE uses stride-four temporal mean pooling, routing dimension 512, and support size K = 16.Source: §4.1 · pp. 4–5

4.2 Multi-Stage Training

  • Training proceeds from interface-only alignment to encoder training and then joint fine-tuning of the encoder, interface, and decoder blocks.The text embedding table and language-modeling head remain frozen throughout all three stages.Source: §4.2 · p. 5

4.3.2 Baseline Models

  • The complete-system comparison covers representative open-weight speech, audio, and omni models whose backbones, data, scales, and adaptation methods differ.The comparison therefore ranks complete systems rather than isolating interface effects.Source: §4.3.2 · p. 5

4.3.3 Benchmarks

  • VoiceBench evaluates general speech capabilities across eleven reported subsets, with Overall defined as their equal-weight mean.The 2B comparisons assess each model with both spoken and written inputs.Source: §4.3.3 · p. 5
  • LibriSpeech test-clean and test-other measure corpus-level word error rate, while MMAU Test-mini measures sound, music, and speech understanding and reasoning accuracy.The evaluation reports each LibriSpeech split and MMAU category alongside the MMAU average.Source: §4.3.3 · p. 5

5 Experimental Results

The experiments compare ConvexLLM with open-weight audio models, compare C-GATE, MLP, and Q-Former under a shared Stage 3 recipe, and vary C-GATE’s support size. ConvexLLM leads the eleven-task VoiceBench aggregate, while interface and support-size results show task-dependent tradeoffs.

5.1 Overall Task Performance

  • 75.36 is ConvexLLM’s VoiceBench Overall, the highest in Table 1, ahead of MiniCPM-o 4.5 at 73.04 and Kimi-Audio at 70.26.Overall is the equal-weight mean of eleven VoiceBench tasks and excludes MMAU and ASR.Source: §5.1 · p. 5, Table 1 · p. 6
  • 90.33 on OpenBookQA, 72.32 on MMSU, 73.42 on SpeechMMLU, and 80.40 on BBH are ConvexLLM’s highest scores in those columns.Its BBH score exceeds MiMo-Audio’s 71.60 by 8.80 points; on open-response tasks, it ranks first on WildVoice at 69.10 but second to MiniCPM-o 4.5 on AlpacaEval, AlpacaEval-Full, and CommonEval.Source: §5.1 · p. 6
  • 62.50% is ConvexLLM’s average MMAU Test-mini accuracy, while its LibriSpeech WER is 2.39% on test-clean and 5.07% on test-other.On MMAU, MiMo-Audio scores 76.63%; on LibriSpeech, Kimi-Audio leads with 1.40% and 2.59% WER, while ConvexLLM beats MiMo-Audio.Source: §5.1 · p. 6

5.2 Comparison of Interfaces

  • 60.35 is C-GATE’s spoken VoiceBench Overall, above MLP at 58.80 and Q-Former at 54.19; its written Overall is 63.20, close to MLP’s 62.72.C-GATE has the smallest written–spoken gap at 2.86 points, versus 3.92 for MLP and 7.44 for Q-Former; MLP leads on LibriSpeech WER and Q-Former on MMAU accuracy.Source: §5.2 · pp. 6–7, Table 2 · p. 7

5.3 Comparison of Top-K Configurations

  • From K = 4 to K = 64, WER decreases and OpenBookQA, IFEval, SD-QA, and VoiceBench Overall increase at every step, but MMAU peaks at K = 16.At K = 64, MMAU averages 58.50%, below K = 16 at 59.40%, despite further improvements in WER and VoiceBench Overall; K = 1 has 86%+ WER on both LibriSpeech splits and 22.30 VoiceBench Overall.Source: §5.3 · p. 7

6 Analysis

The representation analysis compares speech and written inputs for C-GATE, MLP, and Q-Former. C-GATE inputs align more closely with text embedding directions, reach spoken–written alignment earlier in the decoder, and remain at text-like representation scale.

Input Geometry

  • 28.4% of C-GATE input variation lies in the first 100 text directions, compared with 23.7% for written tokens, 8.8% for MLP, and 4.9% for Q-Former.In a projection of 20 questions, C-GATE inputs spread across the region occupied by text tokens, while MLP and Q-Former inputs cluster tightly.Source: §6 · p. 7

Alignment Across Decoder Blocks

  • C-GATE’s spoken–written question cosine similarity reaches 0.69 by decoder block 7 and remains between 0.69 and 0.73 through block 19.At block 7, MLP and Q-Former reach 0.45 and 0.44; MLP reaches 0.67 only at the final block, while Q-Former peaks at 0.63.Source: §6 · p. 7

Representation Scale

  • From the first decoder block onward, C-GATE speech representations follow written-question norms; MLP and Q-Former inputs enter at norms 36.7 and 25.2, versus 0.65 for text embeddings.Text representations reach the MLP/Q-Former input scale only in the last few decoder blocks.Source: §6 · p. 7

7 Conclusion

ConvexLLM represents audio as continuous convex combinations of frozen text embeddings. It leads the compared models on VoiceBench overall, while a controlled 2B interface comparison finds task-dependent tradeoffs; the results support this interface as viable for general-purpose speech and audio models.

7 Conclusion

  • ConvexLLM leads the compared models on the eleven-task VoiceBench aggregate, with its largest margins on spoken knowledge and reasoning, while the strongest baselines lead on MMAU and transcription.In a controlled 2B comparison, C-GATE has the highest aggregate spoken-language performance, text-task performance close to MLP, and the smallest written–spoken gap; MLP leads on transcription and Q-Former on MMAU accuracy.Source: §7 · pp. 7–8
  • ConvexLLM represents audio as continuous convex combinations of frozen text embeddings.The model is presented as a general-purpose speech and audio language model.Source: §7 · p. 7
  • The results support convex combinations of text embeddings as a viable interface for general-purpose speech and audio language models.Source: §7 · p. 8

Limitations

The main comparison evaluates complete systems rather than isolating interface effects.

Limitations

  • The main comparison spans systems that differ in scale, training data, and adaptation recipe, so it compares complete systems rather than interfaces in isolation.Source: Limitations · p. 8

Ethical Considerations

The paper notes privacy and safety concerns for speech systems and records implementation details for the model and its 2B interface comparisons.

Ethical Considerations

  • Speech data may contain identifying or sensitive information, and biases may cause uneven performance across accents and speaker groups, which the authors did not systematically evaluate.The model may also produce incorrect or harmful responses.Source: Ethical Considerations · pp. 8–9

A Implementation and Training Details

  • The appendix documents 9B training, 2B interface and Top-K comparisons, training data, evaluation, and representation measurements.It supplements the architecture and three-stage training description in Section 4.Source: A Implementation and Training Details · p. 10

A.1 Architecture and Parameter Adaptation

  • Routing scores are clipped to [−30, 30]; chunking the 2,048-row vocabulary preserves the all-at-once support while using less memory.Stage 1 freezes encoder and decoder while gradients pass through the decoder to the interface; Stage 2 additionally trains the encoder.Source: §A.1 · p. 11

A.2 Main 9B Training

  • Stages 1 and 2 run 7,000 updates each; Stage 3 is evaluated at 7,500 updates, with transcription and ASR-instruction sampling increased after update 5,000.Stage 3 uses text-teacher distillation for audio examples with written forms and cross-entropy for other audio examples and text replay; training uses 16 H100 GPUs.Source: §A.2 · p. 11

A.3 2B Interfaces and Training Objective

  • The 2B comparison matches the acoustic encoder, decoder, data, and shared training settings across C-GATE, MLP, and Q-Former, whose interface parameter counts differ by less than 0.17%.The interfaces receive 5,000 Stage 1, 7,000 Stage 2, and 18,000 Stage 3 updates; Q-Former produces 64 decoder-input positions, while MLP and C-GATE preserve pooled acoustic-sequence length.Source: §A.3 · p. 11

A.4 Recognition Differences

The recognition comparison finds a C-GATE–MLP tradeoff between transcription and spoken-language tasks, while the compared models also differ in training. The 2B comparison draws on multiple families of speech and text data.

A.4 Recognition Differences

  • The 9B model’s lower WER than 2B C-GATE is 0.08 percentage points on test-clean and 0.47 on test-other, but the models differ in Stage 3 raw-ASR training.The 9B model uses a frozen text teacher, whereas the 2B models use cross-entropy.Source: §A.4 · p. 11
  • C-GATE makes 192 more word edits than MLP on LibriSpeech test-clean and 497 more on test-other; substitutions account for 559 of the 689 additional edits.The respective WER differences are 0.36 and 0.94 percentage points, with paired speaker-bootstrap 95% intervals of [0.24, 0.48] and [0.73, 1.19].Source: §A.4 · pp. 11–12
  • OpenBookQA, MMSU, and SpeechMMLU contribute 1.12 of the 1.54 aggregate VoiceBench points separating C-GATE and MLP.The respective task differences are 6.81, 2.64, and 2.89 points, with paired-item 95% intervals of [1.98, 11.65], [0.68, 4.55], and [1.71, 4.07].Source: §A.4 · p. 12

B.1 Data Composition

  • The 2B comparison groups 45 training sources into recognition, ASR-derived instruction, audio-understanding and synthesized-speech instruction, and text-replay families.The main 9B model uses the same task families; the 2B comparison uses the 9B model’s final Stage 3 sampling proportions throughout its own Stage 3.Source: §B.1 · p. 12

B.2 Training Data Scale

The appendix bounds ConvexLLM’s adaptation audio and contrasts reported training scales, while describing input-length and held-out-partition handling.

B.2 Training Data Scale

  • Counting every adaptation example as audio at the 30-second input cap bounds ConvexLLM’s adaptation audio at 0.271M hours.The stages use 14.3M audio examples each in Stages 1 and 2 and 3.8M examples in Stage 3; the example pool contains 57.2M audio annotations and 0.44M text examples.Source: §B.2 · p. 12
  • Kimi-Audio and MiMo-Audio report audio training scales more than an order of magnitude above ConvexLLM’s adaptation bound, while ConvexLLM obtains the higher VoiceBench aggregate.The comparison notes that reported scales cover different phases and that some baseline totals are absent or imprecise.Source: §B.2 · p. 12, Table 5 · p. 13

B.3 Preprocessing and Held-Out Partitions

  • All training inputs are limited to 30 seconds, and acoustic encoder outputs use temporal mean pooling with stride four.The 2B FLEURS source combines English training and validation partitions and reserves the test partition for evaluation.Source: §B.3 · p. 13

C.1 Benchmarks and Score Aggregation

The appendix specifies the evaluation tasks and procedures, then documents the additional comparisons and representation analyses used to characterize the 2B interfaces.

C.1 Benchmarks and Score Aggregation

  • The main evaluation covers eleven VoiceBench subsets, MMAU Test-mini, and LibriSpeech test-clean and test-other; VoiceBench Overall is the equal-weight mean of eleven subset scores.Open-response scores are GPT-5.4 quality ratings rescaled to 0–100; SD-QA uses judged correctness, AdvBench refusal matching, and five tasks use standard automatic scoring.Source: §C.1 · p. 13

C.3 Additional 2B Speech Tasks

  • For the 2B comparison’s eight additional speech tasks, C-GATE leads on MTBench and all five OpenAudioBench tasks, while MLP has the lowest WER on Common Voice and FLEURS.The additional task results use response-alias matching for OpenAudioBench ReasoningQA and GPT-5.4 judgments for its other tasks.Source: §C.3 · p. 14, Table 7 · p. 15

C.4 Text and Speech Input Conditions

  • The 2B evaluation pairs text inputs, which bypass the encoder and interface, with speech inputs that additionally rely on those components.Baselines use released checkpoints and inference settings; empty responses receive zero or a No decision, and judge-filtered examples are excluded.Source: §C.4 · p. 14, §C.5 · p. 14, §C.6 · p. 14

D.1 Paired Questions and Hidden-State Extraction

  • The representation analysis runs each model on 200 spoken questions and their written counterparts, extracting the input span and each of 24 decoder-block spans before final normalization.The spoken inputs contain audio without a written question, and the analysis excludes surrounding prompt tokens.Source: §D.1 · pp. 14–15

D.2 Energy in Text Principal Directions

  • The text-principal-direction curves measure cumulative energy in centered interface or written-question representations; the measure captures directional variation and is unchanged by common rescaling.Each representation set is centered separately, removing its mean displacement from the text embeddings.Source: §D.2 · p. 15

D.3 Paired Alignment and Representation Norms

  • For paired alignment, each question’s pooled vector has the other 199-question mean subtracted before same-question speech–text cosine similarity is computed; randomized-pair similarities lie within 0.03 of zero at 95% of blocks and models.The norm analysis averages position-level L2 norms within each question and then across questions, giving each question equal weight.Source: §D.3 · p. 15

D.4 Normalized Input PCA Visualization

  • Figure 5 projects unit-normalized input vectors onto the first two text principal components using a shared text mean and axes; they explain 16.12% and 7.64% of normalized text-token variance.The visualization uses uniformly spaced audio positions, with the same count across interfaces, capped at 32.Source: §D.4 · p. 15

E Support Size and Computational Cost

  • Top-K routing limits each position to K embedding rows, and normalized mixture coefficients provide K −1 continuous degrees of freedom.With K = 1, the softmax is constant and gives no router gradient, so that run uses a straight-through estimator with a highest-scoring-embedding forward pass and softmax-average backward gradient.Source: E Support Size and Computational Cost · pp. 15–16, Table 8 · p. 16

Related papers

Loading 2610.23539v1…