Covo-Audio Technical Report

• The motivating goal is to harmonize deep understanding and reasoning, natural expression of paralinguistic and emotional cues, and low-latency, fluent full-duplex interaction.The introduction frames these as qualities current speech-interaction approaches often force systems to compromise among.Source: §1 · p. 1
• The introduction identifies cascaded ASR–LLM–TTS systems and Thinker-Talker models as trade-offs: information loss and error propagation, reduced direct speech instruction following, and harder full-duplex dynamics.The paper adopts interleaved text-speech decoding as its backbone, describing it as more adherent to the oracle LLM philosophy.Source: §1 · pp. 1–2
• Covo-Audio’s 7B model achieves strong or competitive results across speech and audio tasks, outperforming similarly sized models and matching larger systems on multiple key metrics.The reported evaluation spans speech understanding, audio question answering, and half- and full-duplex speech-to-speech dialogue.Source: Figure 1 · p. 2, §1 · pp. 3–4
• Hierarchical tri-modal interleaving combines continuous acoustic features, discrete speech tokens, and text, aligning phrase-level details while preserving sentence-level coherence.The design aims to capture acoustic and prosodic nuances alongside semantic structure in a sequence compatible with LLM generation.Source: §1 · p. 3
• Multi-speaker training and contextual voice adaptation decouple speaker identity from dialogue intelligence, using masked-text-loss pseudo-conversations made from TTS recordings.This approach aims to preserve reasoning while enabling high-fidelity, flexible voice customization with less elaborate dialogue-data preparation; synthesized dialogue can limit voice expressiveness.Source: §1 · pp. 2–3
• Covo-Audio-Chat-FD adds full-duplex interaction during pretraining and supports smooth turn-taking, pause handling, barge-in, and backchanneling.It uses continuous input and discrete output streams without requiring fine-grained text-speech alignment, while maintaining competitive performance with the half-duplex model.Source: §1 · p. 3

2.1 Architecture

• Covo-Audio combines an audio encoder, LLM, speech tokenizer, and decoder to process interleaved audio and text, generate unified text-audio outputs, and carry prior turns as context.The audio adapter reduces the encoder’s 50 Hz output to 6.25 Hz; the tokenizer produces 25 Hz discrete tokens, and a two-stage decoder reconstructs 24K waveforms.Source: §2.1 · pp. 4–5

2.2 Pre-training

• Two-stage pretraining bridges speech and text, then jointly optimizes the adapter and LLM with multimodal tasks, including full-duplex training and hierarchical tri-modal interleaving.Stage 1 freezes the encoder and LLM while training the adapter on multilingual ASR; Stage 2 uses utterance- and phrase-level interleaving to preserve long-form semantic context.Source: §2.2 · pp. 5–6

2.3 Speech-to-Speech Spoken Dialogue

• Covo-Audio-Chat is trained across text-to-text, text-to-audio, audio-to-text, and audio-to-audio tasks using parallel data, conversational rewrites, and emotion-aware dialogues.The approach starts from 10M text instructions, transfers them to audio tasks using synthesized speech and KL-based distillation, and adds colloquial and seven-category empathy data.Source: §2.3 · pp. 6–7

2.4 Intelligence-Speaker Decoupling Technique

• Intelligence-speaker decoupling converts TTS data into dialogue-form training examples while excluding text responses from the loss, enabling voice customization with lightweight TTS data.The method aims to transfer speaker naturalness while preserving dialogue-level intellectual ability despite format differences between TTS and conversational data.Source: §2.4 · pp. 7–8

2.5 Full-Duplex Voice Interaction

• Covo-Audio-Chat-FD enables simultaneous listening and speaking through chunked dual-stream input, turn-control tokens, and training data that includes barge-ins and backchanneling.The model emits THINK while listening, SHIFT when beginning to speak, and BREAK to stop speaking and return to listening.Source: §2.5 · pp. 8–9

2.6 Speech Understanding

• Speech-understanding training covers linguistic content and speaker or emotional attributes, using cross-modal examples and demographic stratification to support modality-invariant recognition and mitigate data bias.The framework mixes text and speech query-response configurations and prioritizes demographic balancing in the order age, language, then gender.Source: §2.6 · pp. 9–10

2.7 Audio Understanding

• Audio-understanding training combines foundational acoustic perception with chain-of-thought data that separates reasoning from answers, allowing the model to articulate its reasoning about audio.The reasoning set merges AudioMCQ and AF-Think into 1M instances, with missing AF-Think answers generated and formatted using DeepSeek V3.1.Source: §2.7 · pp. 10–11
• GRPO further refines audio reasoning using rewards for answer accuracy, output format, reasoning-answer consistency, and thinking quality.Thinking quality evaluates logical soundness, factual correctness, error identification, language coherence, and conciseness.Source: §2.7 · p. 11

3.1 Evaluation on Pre-training

• Pre-training achieves parity with specialized A2A models on story continuation and significantly outperforms baselines on A2A reasoning and grammatical accuracy, plus A2T and T2T consistency.The model is also competitive with GLM-4-Voice-Base on ASR and TTS despite being a unified multimodal system; the evaluation suite includes A2A, A2T, T2T, ASR, and TTS tasks.Source: Table 3 · p. 12, Table 4 · p. 12, §3.1 · p. 13

3.2 Evaluation on Speech-to-Speech Spoken Dialogue

• Covo-Audio-Chat leads Chinese URO-Bench reasoning and spoken-dialogue tasks, while showing top-tier comprehension scores and best English Gsm8kEval performance.Chinese scores include SQuAD 77.34, OpenbookQA 83.60, APE 68.42, MLC 80.69, AlpacaEval 90.02, and Wildchat 90.41; Repeat is 98.17 and HSK5 is 88.00. On VCB Bench, it leads text instruction following (TIF: 93.07; TIF-En: 89.94), multi-turn dialogue (MTD: 87.70), and robustness sub-tasks (SV: 88.94; EV: 87.13; CV: 90.37), while GK and DC are modest versus Qwen3-Omni.Source: §3.2 · pp. 13–14, Table 5 · p. 14, Table 6 · p. 15
• Covo-Audio-Chat achieves Mandarin empathy SOTA scores for anger (4.89), sadness (4.93), and anxiety (5.00), while TTS voice transfer preserves comparable bilingual dialogue performance.The empathy results use LLM-as-a-Judge and may prioritize semantic content over voice expression; a preliminary subjective test found performance below top-tier productions such as Doubao4. The decoupling technique enables flexible voice customization with lightweight TTS data.Source: Table 5 · p. 14, Table 7 · p. 15, §3.2 · pp. 15–16

3.3 Evaluation on Full-Duplex Interaction

• Covo-Audio-Chat-FD substantially outperforms Moshi and Freeze-Omni across English and Chinese understanding, reasoning, and oral-conversation tasks while maintaining comparable dialogue performance to Covo-Audio-Chat.It achieves 96.81% success handling user interruption and 93.89% handling backchanneling; occasional premature responses during short pauses contribute to a slight dialogue-performance drop and correlate with pause-handling success.Source: Table 8 · p. 16, §3.3 · pp. 16–17, Table 9 · p. 16

3.4 Evaluation on Speech Understanding

• Under identical decoding and data conditions, Covo-Audio achieves competitive ASR performance with an average WER of 4.71 and the best overall AIR-Bench performance across evaluated paralinguistic tasks.On emotion recognition, it achieves absolute gains of over 10–25 percentage points compared to prior methods.Source: §3.4 · p. 17

3.5 Evaluation on Audio Understanding

• On MMAU, Covo-Audio scores 75.30% on average, ranking second among open-source models; on MMSU, it scores 66.64%, the highest among evaluated systems.On MMAU, its 76.05% music-understanding score is highest among all models, and its average is best among comparable-scale 7B models; Step-Audio 2 scores 77.58% with a 30B architecture. On MMSU, Covo-Audio leads perception with 58.95%, while its reasoning score of 74.83% trails Qwen2.5-Omni at 79.83%.Source: Table 11 · p. 18, §3.5 · p. 19

Native Multimodal Models

• Native multimodal models such as Gemini and GPT-4o use shared multimodal vocabularies and hidden spaces, while Emu and Chameleon tokenize modalities within a unified Transformer.These approaches contrast with vision-language systems that append visual features to text-centric models.Source: §4 · p. 20

Large Audio Language Models

• Llama-Omni and Mini-Omni combine speech adapters and non-autoregressive decoders, while Freeze-Omni freezes its LLM backbone to preserve textual knowledge during streaming audio integration.Qwen-Omni separates semantic deliberation and speech generation into Thinker and Talker modules; Qwen3-Omni uses MoE and supports over 40 minutes of audio with real-time multilingual streaming.Source: §4 · p. 20
• Moshi predicts audio and text streams simultaneously for full-duplex interaction, while VITA-Audio reduces first-token latency and Step-Audio2 and MiMo-Audio process interleaved text and audio streams.GLM-4-Voice, Baichuan-Audio, and Baichuan-Omni support Chinese-English conversations with emotional resonance; Step-Audio-R1 uses reasoning-centric RL for non-vocal audio perception and conversational logic.Source: §4 · pp. 20–21

Full-Duplex Spoken Dialogue LALM

• Full-duplex LALMs either use external modules to decide when to listen or speak, or synchronize speech input and output in a dual-stream end-to-end model.Full-duplex interaction supports turn-taking, interruption, and backchanneling.Source: §4 · p. 21

5 Conclusion

• Covo-Audio achieves state-of-the-art or competitive performance across speech-text modeling, spoken dialogue, speech understanding, audio understanding, and full-duplex interaction, rivaling or exceeding comparable-scale models.The report attributes these results to large-scale curated pretraining and targeted post-training.Source: §5 · p. 21
• Covo-Audio-Chat-FD substantially outperforms open-source LALMs such as Moshi and Freeze-Omni on spoken dialogue and full-duplex interaction while maintaining highly comparable dialogue performance to Covo-Audio-Chat.Source: §5 · p. 21
• The pretrained foundation model outperforms GLM-4-Voice-Base on speech-text benchmarks, with results suggesting semantic extraction for high-level reasoning beyond transcription.The cited benchmarks include topic StoryCloze, sBLIMP, sWUGGY, and ASR/TTS.Source: §5 · p. 21
• Covo-Audio-Chat demonstrates spoken dialogue understanding, reasoning, oral conversation, instruction following, and robustness, alongside competitive recognition of nuanced emotional cues and empathetic response generation.The report evaluates dialogue on URO-Bench and VCB Bench and empathetic responses against mainstream spoken dialogue LALMs, including advanced business models.Source: §5 · p. 21
• Results across multiple Speech Understanding and Audio Understanding benchmarks highlight Covo-Audio's general audio question-answering capabilities.Source: §5 · p. 21
• Covo-Audio-Chat-TTS achieves comparable bilingual dialogue performance to Covo-Audio-Chat through intelligence-speaker decoupling, which separates voice rendering from dialogue intelligence and requires only lightweight TTS data for voice customization.Source: §5 · p. 21
• The report presents Covo-Audio's success as evidence that 7B-scale models can combine high semantic reasoning with intricate audio intelligence, and identifies scaling up as future work.The authors attribute this potential to optimized pretraining and post-training strategies.Source: §5 · p. 22

A.1.1 Anger Case

• Covo-Audio-Chat scored 5, matching GPT-4o-mini, GPT-4o, and Qwen2.5-Omni on the anger case.Its reply validates the user’s anger and offers to think through a solution together.Source: §A.1.1 · p. 29
• Doubao also scored 5, while Baichuan-Audio scored 3, Kimi-Audio scored 2, and Step-Audio scored 1.Source: §A.1.1 · p. 29

A.1.2 Anxiety Case

• Covo-Audio-Chat回应楼下疑似小偷的担忧时建议先确认声音方向，并联系物业或报警保障安全，得分5。用户表示刚听到楼下有奇怪声音，担心是小偷。Source: §A.1.2 · p. 29
• Doubao和GPT-4o在该案例也均得分5；Doubao建议查看猫眼或联系物业，GPT-4o列举日常噪声可能并提醒注意安全。Source: §A.1.2 · p. 29
• GPT-4o-mini得分4，建议确认声音来源，并在担忧时通知家人或相关人员求助。Source: §A.1.2 · p. 29
• Baichuan-Audio和Kimi-Audio在该案例均得分1；两者都建议采取安全措施或联系物业、警方等求助。Source: §A.1.2 · p. 30

A.1.3 Joy Case

• Covo-Audio-Chat celebrates the engagement and scores 5, matching Kimi-Audio, Doubao, GPT-4o, and GPT-4o-mini, which also each score 5.Their replies congratulate the user and invite them to share more about the proposal.Source: §A.1.3 · p. 30
• Qwen2.5-Omni and Step-Audio also congratulate the user and each receive Score: 5.Both replies respond to the happy news and ask about the user's plans or proposal details.Source: §A.1.3 · p. 30
• Baichuan-Audio receives Score: 2, responding with an extended explanation of the announcement rather than a conversational congratulation.The user's prompt announces an engagement after a proposal that evening.Source: §A.1.3 · p. 30

A.1.4 Sadness Case

• Covo-Audio-Chat (Score: 5) acknowledges the job seeker’s distress, offers reassurance, and invites them to review their résumé or discuss target roles.The user says they have been submitting résumés without receiving replies.Source: §A.1.4 · p. 30
• GPT-4o (Score: 5) validates the frustration, suggests revising the résumé or cover letter, and offers to discuss the user’s background and interests.The response frames job hunting without replies as emotionally draining.Source: §A.1.4 · p. 30
• Step-Audio (Score: 5) recognizes the challenges of job hunting and asks about target positions and prior strategies to explore new approaches together.It reassures the user that the difficulty may be temporary.Source: §A.1.4 · p. 30
• Qwen2.5-Omni (Score: 5) responds sympathetically, asks what roles the user targets, and suggests checking the résumé and concentrating applications.It also asks whether the roles are especially competitive.Source: §A.1.4 · p. 30
• Doubao (Score: 5) normalizes unanswered applications, suggests hiring-season growth or slow company processes as possibilities, and asks what roles the user applied for.Source: §A.1.4 · pp. 30–31
• Kimi-Audio (Score: 3) briefly recognizes the pressure and effort involved in job hunting and encourages the user not to lose heart.Source: §A.1.4 · pp. 30–31
• Baichuan-Audio (Score: 1) offers résumé tailoring, networking, and multiple job-search channels, but its response includes unclear phrasing.Source: §A.1.4 · pp. 30–31
• GPT-4o-mini (Score: 1) gives a checklist on résumé optimization, tailored cover letters, and LinkedIn networking, with its response ending mid-suggestion.Source: §A.1.4 · pp. 30–31

A.2.1 Anger Case

• The reported scores were 5 for Covo-Audio-Chat, GPT-4o, and GPT-4o-mini; 4 for Doubao and Step-Audio; and 3 for Kimi-Audio.Source: §A.2.1 · p. 31
• Covo-Audio-Chat acknowledged the frustration, suggested checking permissions and subscriptions and canceling suspicious charges, and offered step-by-step support.GPT-4o and GPT-4o-mini also scored 5 while recommending subscription checks and cancellation or customer-support options.Source: §A.2.1 · p. 31
• Lower-scoring responses were less useful: Baichuan-Audio repeated its customer-service suggestion, Qwen2.5-Omni repeated a setting check after the user said it had failed, and Kimi-Audio was incoherent.Baichuan-Audio scored 2, Qwen2.5-Omni scored 1, and Kimi-Audio scored 3; Step-Audio scored 4 but suggested rebooting or reinstalling updates rather than addressing charges directly.Source: §A.2.1 · p. 31

A.2.2 Anxiety Case

• Score 5: Covo-Audio-Chat matches GPT-4o, GPT-4o-mini, and Doubao in the first-flight anxiety exchange.The four systems each receive Score: 5.Source: §A.2.2 · p. 32
• Covo-Audio-Chat validates the user’s nervousness, reframes sweaty palms as excitement, and guides them through deep breaths.The user says they are flying for the first time tomorrow and their palms will not stop sweating.Source: §A.2.2 · p. 32
• Step-Audio scores 3, while Kimi-Audio, Baichuan-Audio, and Qwen2.5-Omni each score 1.Step-Audio responds with questions and enthusiasm; the three score-1 responses include garbled or incoherent content.Source: §A.2.2 · p. 32

A.2.3 Joy Case

• Covo-Audio-Chat celebrated the news, acknowledged the user’s excitement, and asked how they felt, receiving Score 5.The reply framed the wedding date as a special moment and invited the user to share their feelings.Source: §A.2.3 · p. 32
• GPT-4o, GPT-4o-mini, Doubao, Step-Audio, and Baichuan-Audio each received Score 5 for their wedding-announcement replies.Source: §A.2.3 · pp. 32–33
• Kimi-Audio received Score 3, while Qwen2.5-Omni received Score 1.Source: §A.2.3 · p. 33

A.2.4 Sadness Case

• Score 5: Covo-Audio-Chat, GPT-4o, and GPT-4o-mini each acknowledged the disappointment and invited the user to share more.Their replies suggested possible explanations for the emoji without dismissing the user’s feelings.Source: §A.2.4 · p. 33
• Score 5: Doubao and Step-Audio offered sympathy and follow-up invitations, though Doubao characterized the emoji as dismissive.Both responses received the same score as Covo-Audio-Chat and the two GPT-4o variants.Source: §A.2.4 · p. 33
• Score 3: Kimi-Audio and Baichuan-Audio gave less fitting replies, including an unclear clarification question and awkward phrasing.Kimi-Audio asked about a “phone sub,” while Baichuan-Audio referred to “hard feelings on with.”Source: §A.2.4 · p. 33
• Score 1: Qwen2.5-Omni shifted from acknowledging the user’s feelings to asking for a joke.The response also included an unrelated “Human?” utterance.Source: §A.2.4 · p. 33

B Audio Understanding

• Table 12 compares performance on MMAU-v05.15.25, with results for other methods sourced from the official MMAU leaderboard.The supplied caption gives the table’s scope but no result values.Source: Table 12 · p. 34
• Table 13 presents detailed MMAR results, with model results sourced from the original MMAR paper and their respective original publications.The supplied caption does not provide the table’s values or establish a performance winner.Source: Table 13 · p. 34